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Foreword 


The  Armed  Services  Vocational  Aptitude  Battery  (ASVAB)  is  administered  to  all 
individuals  interested  in  enlisting  into  military  service.  The  test  covers  a  range  of  topics, 
including  automotive  and  shop  knowledge,  electronics  information,  mechanical 
reasoning,  verbal  and  mathematical  ability,  and  general  science  information.  Scores  are 
used  to  determine  whether  a  person  qualifies  for  service  as  well  as  the  likelihood  of 
successfully  completing  job-specific  training,  and,  by  extension,  success  on  the  job. 

Once  a  person  is  in  service.  Sailors  may  retake  the  test  to  improve  their  test  scores 
and  qualify  for  a  different  Navy  job.  The  in-service  version  of  the  ASVAB  is  referred  to  as 
the  Armed  Forces  Classification  Test  (AFCT).  It  is  a  retired  paper-and-pencil  version  of 
the  ASVAB.  Responses  are  recorded  on  “bubble”  answer  sheets,  scored  by  hand  using  a 
stencil  overlay.  The  scale  scores  are  calculated  by  summing  up  the  number  of  correct 
items  for  a  subtest. 

Several  things  occurred  to  make  these  procedures  impractical.  First,  the  Navy  is 
actively  reshaping  the  force  structure,  eliminating  some  jobs  and  growing  others.  As  a 
result,  many  Sailors  are  being  asked  to  change  jobs,  which  has  dramatically  increased 
the  number  of  AFCT  administrations.  This  makes  the  long,  slow,  and  error  prone 
process  of  administering,  timing,  and  hand-scoring  tests  impractical.  Secondly,  in  July 
2002,  the  Navy  adopted  a  new  normative  reference  population  to  compare  test  scores 
against.  The  new  national  normative  data  was  collected  in  1997  (the  old  data  was  from 
1980).  There  are  a  number  of  differences  between  the  two  populations  that  substantially 
change  what  a  score  means.  For  example,  a  50  on  the  1980  Math  Knowledge  scale  is 
actually  a  52  on  the  1997  scale;  but  a  50  on  the  1980  Electronics  Information  scale  is 
actually  a  47  on  the  1997  scale.  Thus,  the  AFCT  scores  need  to  be  on  the  current  score 
scale.  However,  there  is  a  further  complication.  The  1997  scores  scales  are  all  based  on 
scoring  algorithms  using  a  3-parameter  logistic  model  from  Item  Response  Theory. 
Practically  speaking,  this  means  that  when  someone  is  hand  scoring  a  test,  each  correct 
item  must  be  scored  using  three  5-decimal  place  numbers  in  an  exponential  function, 
then  summed.  This  is  nearly  impossible  to  do  correctly  by  hand  for  all  225  AFCT  test 
items. 

One  solution  to  these  problems  would  be  to  move  the  AFCT  test  onto  a  computer.  A 
computer-based  AFCT  could  accurately  and  consistently  handle  test  administration 
(e.g.,  each  of  the  9  subtests  are  separately  timed)  and  background  algorithms  could 
instantly  and  perfectly  score  items,  estimate  subtest  scores,  and  place  the  summary 
scores  on  the  1997  scale.  Moreover,  the  volume  of  test  administrations  would  have  no 
impact.  However,  past  research  has  shown  that  “computerizing”  a  paper  test  can 
inadvertently  alter  test  scores  in  both  large  and  small  ways  and  differently  for 
demographic  groups.  This  report  outlines  efforts  to  create  a  computer-based  version  of 
the  AFCT  and  determine  if  there  are  scores  differences  between  the  paper  and  computer 
based  versions. 


DAVID  L.  ALDERTON,  Ph.D. 

Director 
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Summary 


Problem 

Increases  in  the  number  of  in-service  AFCT  administrations,  the  shift  to  the  1997 
score  scale,  and  the  necessity  of  using  Item  Response  Theory  scoring  methods  has  made 
the  current  AFCT  administrative  procedures  and  hand-scoring  methods  impractical. 
Administering  the  AFCT  across  the  Internet  may  shorten  testing  time  and  increase 
security  of  test  items,  and  it  will  reduce  scoring  errors  and  produce  scores  based  on  the 
current  score  scale.  This  is  also  in  line  with  Navy  initiatives  to  move  more  training  and 
testing  to  the  Internet.  However,  the  impact  on  test  scores  from  moving  from  a  paper- 
based  test  to  computer-based  administration  is  unknown. 

Objective 

The  goal  of  this  study  is  to  create  a  web-based  version  of  the  AFCT,  then  administer 
both  the  paper-  and  web-based  versions  to  personnel  and  determine  if  there  are 
substantial  differences  between  test  scores  across  the  two  modes  of  administration. 

Approach 

Students  at  Naval  Service  Training  Command,  Great  Lakes,  were  assigned  to  take 
either  the  paper-  or  web-based  version  based  upon  the  last  digit  of  their  Social  Security 
Number;  those  with  even  numbers  were  given  the  paper  version  and  those  with  odd 
numbers  were  given  the  web  version.  Tests  were  scored  and  a  scale  score  was  created  for 
each  respondent.  Scale  scores  were  compared  for  both  versions  overall  as  well  as  by 
gender  and  ethnicity  (majority  vs.  minority). 

Findings 

1.  Web  AFCT  as  programmed  was  stable;  the  only  issue  was  related  to  lost 
connectivity  to  the  server. 

2.  Paper-based  scores  tended  to  be  slightly  higher,  especially  for  Paragraph 
Comprehension  and  Assembling  Objects. 

3.  Larger  differences  between  versions  were  seen  for  women  than  men,  possibly  due 
to  the  small  number  of  women  participants. 

4.  Both  majority  and  minority  groups  showed  differences  in  scores  obtained  from 
the  two  versions,  and  a  similar  pattern  was  shown  in  the  overall  findings. 


IX 


Recommendations 

1.  AFCT  can  be  administered  across  the  web,  but  a  concept  of  operations  for  Fleet¬ 
wide  administration  needs  to  be  determined.  For  example: 

a.  Where  should  the  test  software  be  hosted? 

b.  How  can  a  version  of  the  test  be  made  available  to  deployed  ships  and 
submarines? 

2.  Determine  if  small  point  differences  between  the  two  versions  can  be  considered 
negligible. 

3.  Consider  improvements  to  web-based  layout,  such  as: 

a.  Increased  font  size  and  different  fonts  for  Paragraph  Comprehension 
display. 

b.  Find  way  to  allow  web-based  respondents  to  “write”  on  the  displayed 
graphics  for  Mechanical  Comprehension  and  Assembling  Objects  subtests. 

4.  Replicate  equating  study  with  larger  group. 
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I  ntroduction 


Problem  and  Background 

Before  entering  the  Navy,  potential  recruits  complete  the  Armed  Services  Vocational 
Aptitude  Battery  (ASVAB).  This  test  consists  of  nine  subtests  covering  a  wide  variety  of 
subjects  as  shown  in  Table  i.  If  not  taking  a  computerized,  adaptive  version  of  the 
ASVAB,  each  subtest  has  a  specific  number  of  questions  and  a  time  limit,  also  indicated 
in  Table  i. 


Table  1 

ASVAB  Subtests 


Subtest 

Topics 

Number  of 
questions 

Time 

Limit 

General  Science  (GS) 

Life  science,  earth  and  space  science, 
and  physical  science 

25 

11 

Arithmetic  Reasoning 
(AR) 

Arithmetic  word  problems 

30 

36 

Word  Knowledge  (WK) 

Synonyms 

35 

11 

Paragraph 

Comprehension  (PC) 

Extracting  information  from 
paragraphs 

15 

13 

Mathematics  Knowledge 
(MK) 

Mathematical  concepts  and 
applications 

25 

24 

Electronics  Information 
(El) 

Electrical  current,  circuits,  devices, 
and  electronic  systems 

20 

9 

Automotive  and  Shop 
Information  (AS) 

Automotive  maintenance  and  repair, 
wood  and  metal  shop  practices 

25 

11 

Mechanical 
Comprehension  (MC) 

Mechanical  devices,  structural 
support,  and  properties  of 
materials 

25 

19 

Assembling  Objects 
(AO) 

Spatial  visualization 

25 

15 

The  military  service  branches  may  opt  to  include  additional  subtests.  Previous  tests 
have  included  Coding  Speed  (finding  information  in  a  table  quickly)  and  Numerical 
Operations  (rapidly  computing  simple  mathematical  calculations). 

The  ASVAB  is  administered  at  Military  Entrancing  Processing  Stations  (MEPS)  as 
part  of  the  formal  processing  for  acceptance  into  any  enlisted  military  position.  The  test 
results  will  be  used  to  intellectually  qualify  an  applicant  for  service  and  then  used  by 
classifiers  to  determine  an  applicant’s  suitability  for  a  specific  job.  Within  the  Navy,  this 
classification  to  jobs— or  Navy  ratings— is  based  upon  optimally  formed  composites  of 
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ASVAB  subtests  and  “cut-scores”^  which  have  been  found  to  predict  success  in  job 
training  schools.  The  ASVAB  composites  and  cut-scores  are  periodically  re-evaluated  to 
ensure  that  the  most  optimal  ASVAB  standard  is  used  for  each  school  (see,  e.g.,  Held  & 
Monzon,  1991). 

Once  in  the  military,  a  version  of  the  ASVAB  called  the  Armed  Forces  Classification 
Test  (AFCT)  may  be  administered  to  personnel  who  are  interested  in  changing  their  job, 
allowing  personnel  the  opportunity  to  improve  their  scores  to  qualify  for  their  desired 
job.  The  AFCT  is  a  retired  version  of  the  ASVAB.  At  present,  it  is  only  available  in  paper 
format,  which  in  normal  operational  settings  means  it  must  be  hand  scored.  However, 
today’s  ASVAB  subtests  are  exclusively  scored  using  a  3-parameter  logistics  (3PL)  Item 
Research  Theory  (IRT)  model  which  increases  the  precision  of  measurement.  Practically 
speaking,  this  means  that  when  someone  is  hand  scoring  a  test,  each  correct  item  must 
be  scored  using  three  5-decimal  place  numbers  in  an  exponential  function,  then 
summed.  This  is  nearly  impossible  to  do  correctly  by  hand  for  all  225  AFCT  test  items. 
(IRT  scoring  was  used  in  this  project  for  both  the  paper-based  and  computer-based 
versions.) 

Objective 

This  report  describes  an  effort  undertaken  to  create  a  web-based  AFCT  and 
determine  if  there  were  differences  in  scores  between  the  paper-  and  web-based 
versions.  Of  particular  interest  were  comparisons  of  scores  for  males,  females,  majority- 
ethnicity  personnel,  and  minority-ethnicity  personnel. 


Approach 


Test  I  nstruments 

The  paper  AFCT  booklet  contains  4  to  10  questions  per  page,  depending  upon  the 
subtest.  Examinees  mark  their  selected  answer  on  a  standard  scantron  (“bubble”) 
answer  sheet.  Examinees  are  not  allowed  to  continue  to  subsequent  subtests  until  time 
is  up  or  until  test  administrators  determined  by  a  show  of  hands  that  all  examinees  had 
completed  a  subtest.  All  testing  materials  were  collected  before  examinees  were 
dismissed. 

The  web-based  version  was  programmed  using  WebQuiz  XP  (SmartLite  Software, 
Inc.).  Each  subtest  was  created  individually.  All  questions  were  typed  into  WebQuiz  XP 
and  then  verified  by  two  individuals  for  accuracy.  Some  subtests  (especially  MC  and  AO) 
contained  graphics;  these  graphics  were  scanned  using  a  OneTouch  8900  scanner. 

Every  graphic  was  cleaned  in  Photoshop  4  to  remove  any  imperfections  and  then  saved 
as  a  .png  at  96  dpi.  Their  display  sizes  were  the  same  as  in  the  paper  test  booklets. 


1  Classification  composites  are  simple  summations  of  individual  subtest  scores.  If,  for  example,  a 
composite  consists  of  AR+MC,  and  a  person  scored  52  on  AR  and  61  on  MC,  their  composite  score  would 
be  113.  If  the  “cut-score”  for  qualifying  for  a  particular  school  is  AR+MC  =  111,  then  this  person  would 
“make  the  cut”  and  be  deemed  qualified  for  that  school,  based  on  that  cut-score. 
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WebQuiz  XP  created  .mdb  files  for  the  questions  and  the  data,  and  additional  .asp 
files  for  the  actual  display  of  the  questions.  Before  viewing  even  the  instructions  for  the 
first  subtest,  an  administrative  password  was  required  to  ensure  that  everyone  started 
after  the  verbal  instructions  were  given.  At  the  bottom  of  the  instructions  of  the  first 
subtest,  an  SSN  was  required  to  continue  to  the  first  question.  The  .asp  files  for  the 
succeeding  subtests  were  modified  so  that  this  SSN  was  required  to  begin  and 
propagated  throughout  each  of  the  subtests.  However,  the  SSN  was  “lost”  once  the  next 
subtest  had  begun,  ensuring  that  the  respondent  could  not  go  back  to  the  answers  in  the 
previous  subtest  without  having  to  return  to  the  beginning  of  the  AFCT  and  reenter  an 
SSN. 

Each  subtest  of  the  web-based  version  included  the  same  instructions  and  sample 
questions  as  in  the  paper  condition  with  just  minor  wording  changes  (e.g.,  “Mark  on 
your  answer  sheet”  became  “Click  the  answer”).  The  instructions  were  on-screen  and 
read  silently  by  each  examinee,  whereas  they  were  read  aloud  by  a  proctor  for  the  paper- 
based  version.  The  instructions  were  not  timed;  the  timer  would  start  as  soon  as  the  first 
question  appeared  on  screen,  similar  to  the  paper  process  when  the  timer  did  not  start 
until  turning  to  the  first  question. 

Figure  i  shows  the  general  screen  layout  of  each  question  in  the  web-based 
version.  The  title  of  the  subtest  was  displayed  under  the  Internet  Explorer  toolbars,  the 
question  and  answer  choices  located  directly  below  that,  and  the  timer  was  in  the 
bottom  left  corner  of  the  window.  Only  one  question  was  presented  at  a  time.  Within 
each  subtest,  the  examinee  could  move  forward  or  backward  through  the  subtest  using 
the  “Next”  and  “Previous”  buttons  (previous  not  available  on  question  i  of  each  subtest); 
moving  to  a  different  question  would  retain  the  person’s  selected  answer.  Moving 
through  questions  did  not  impact  the  timer;  it  would  continue  counting  down  the 
remaining  time. 
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Figure  1.  Web- based  AFCT  screen  layout. 

For  the  Paragraph  Comprehension  subtest,  each  paragraph  relates  to  a  few 
subsequent  questions.  In  the  web-based  version,  the  paragraph  was  presented  with  the 
first  question;  succeeding  questions  were  presented  on  separate  pages  but  the  examinee 
could  go  back  to  the  paragraph  using  the  “Previous”  button. 

Unlike  the  paper-based  version,  the  web-based  AFCT  was  self-paced.  If  one  subtest 
took  only  lo  minutes  for  a  respondent  to  complete,  they  could  then  proceed  to  the  next 
subtest  when  they  were  ready.  If  the  respondent  exceeded  the  total  time  on  a  subtest, 
the  subtest  would  stop  and  indicate  that  it  was  now  time  to  continue  to  the  next  subtest. 

Testing  Room  Layout 

Paper-based  testing  occurred  in  one  of  two  locations.  The  first  location  was  a 
classroom,  with  each  examinee  having  an  individual  table/chair  combination.  To 
minimize  concerns  about  cheating,  each  column  of  chairs  was  about  2  feet  apart  from 
the  one  next  to  it. 

The  second  location  for  paper-based  testing  was  a  hangar  bay.  Large  tables  were 
available,  with  two  chairs  per  table.  Respondents  were  asked  to  sit  in  every  other  chair, 
to  again  minimize  the  potential  for  cheating. 

Because  of  the  electricity  requirements  for  the  web-based  testing  condition,  testing 
only  occurred  in  a  classroom  environment.  The  chairs  were  arranged  as  seen  in  Figure  2, 
with  pairs  of  subjects  facing  each  other.  Each  chair  was  2  feet  from  other  chairs  to  the 
left  and  right,  and  the  laptop  screens  served  as  dividers  between  examinees. 
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Figure  2.  Room  layout  for  web- based  testing. 

Testing  computers  were  Compaq  Evo  N1020V  laptops  with  Windows  XP  and  IE  6.x 
loaded.  Up  to  three  servers  (either  Windows  NT  or  2003)  were  used  at  a  time,  with  at 
most  23  testing  laptops  connected  through  CAT-5  cables  to  each  server.  The  Windows 
2003  server  took  longer  to  present  succeeding  pages,  but  the  loading  time  did  not  count 
against  the  subtest  time  as  there  was  no  question  displayed  during  the  loading  period. 

In  both  conditions,  at  least  two  proctors  monitored  testing.  In  the  classroom 
conditions,  proctors  wandered  between  each  column  of  desks;  in  the  web-based 
condition,  proctors  also  wandered  between  each  row  of  desks  and  were  usually  facing 
opposite  directions  to  best  monitor  test-taking. 

Subjects 

Testing  was  conducted  from  31  July  to  13  September  2006.  Participants  were 
personnel  assigned  to  NSTC  Great  Lakes  during  that  time  period.  The  majority  were 
students  in  “A”  School  (who  were  selected  by  duty  section)  or  in  indoctrination  sections 
prior  to  “A”  School  (generally  after  their  last  indoctrination  class  ended).  A  small 
portion  was  students  not  under  instruction  (NUI)  and  two  were  staff  members  (E-6). 

Results  from  subjects  who  did  not  include  a  complete  SSN  on  testing  paperwork  or 
who  did  not  sign  the  Privacy  Act  Statement  were  not  included  in  subsequent  analyses.  In 
addition,  the  two  staff  members  were  not  included  because  of  the  potential  that  their 
additional  Navy  service  could  have  changed  their  responses. 

A  few  participants  took  the  test  multiple  times,  in  the  same  or  different  format.  Only 
the  first  set  of  answers  was  used  for  analyses.  In  one  case,  one  format  was  complete  and 
the  second  only  had  a  few  answers;  the  assumption  was  that  the  person  realized  they 
were  doing  it  again  and  stopped,  so  only  the  complete  set  of  answers  was  used.  In 
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another  case,  both  sets  of  answers  were  complete  and  it  was  unclear  which  occurred 
first;  all  data  for  that  person  was  excluded.  Table  2  contains  the  overall  number  of 
respondents  for  each  of  the  analysis  groups. 


Table  2 

Overall  number  of  AFCT  subjects 


Overall 

Male 

Female 

Majority 

Minority 

Paper (p) 

689 

547 

91 

412 

177 

Web  (w) 

762 

572 

111 

361 

229 

Some  subjects  seemingly  chose  to  not  take  the  test  seriously  and  recorded  some 
pattern  (e.g.,  all  As,  ABCD  repeated,  AABBCCDD  repeated,  etc.)  in  their  answers. 
Because  it  is  unclear  what  constitutes  a  pattern  and  what  is  a  legitimate  answer,  and 
because  this  behavior  likely  occurred  in  both  paper-based  and  web-based  modes,  all 
these  answers  are  included. 

Some  of  the  subtests  were  partially  blank.  This  may  have  occurred  because  they 
skipped  questions,  ran  out  of  time,  or  had  to  leave  the  test  early.  Data  were  retained  for 
those  who  had  answered  at  least  half  of  the  questions  in  the  subtest.  The  number  of 
subjects  varies  by  subtest;  Ns  are  included  where  appropriate  in  the  results  section. 

Procedure 

As  participants  arrived,  all  were  asked  to  sign  in.  The  sign-in  sheets  included  spaces 
for  names,  the  last  4  digits  of  their  Social  Security  Numbers,  gender,  ethnicity,  and 
rating  or  school.  Once  signed  in,  participants  generally  were  divided  by  the  last  digit  of 
their  SSN,  with  even  SSNs  receiving  the  paper  AFCT  and  odd  SSNs  receiving  the  web- 
based  AFCT.  There  were  a  few  testing  sessions  where  there  was  insufficient  room  for 
paper-based  testing;  in  those  cases,  respondents  were  first  escorted  into  the  computer 
lab  until  all  computers  were  filled,  and  then  the  overflow  went  to  another  room  to 
receive  the  paper-based  test.  (See  Appendix  A  for  details  of  additional  analyses 
conducted  to  ascertain  if  the  two  groups  were  similar  based  upon  their  pre-existing 
ASVAB  scores.) 

To  minimize  the  impact  of  computer  illiteracy  on  the  web-based  testing,  examinees 
in  the  web-based  condition  were  allowed  to  play  games  before  the  testing  started  using 
the  koala-pad  mouse  on  the  laptop.  A  colored  dot  was  affixed  to  each  computer  to 
indicate  which  button  was  the  left  mouse  button  to  be  used  to  select  answers. 

General  AFCT  Form  19G  instructions  were  adapted  for  the  experiment.  Appendix  B 
includes  the  modifications  to  the  general  instructions,  with  annotations  provided  to 
indicate  where  the  additions  occurred  and  for  whom. 

In  both  conditions,  respondents  were  read  instructions  about  the  voluntary  nature  of 
the  testing.  In  the  course  of  the  instructions,  they  were  then  asked  to  read  and  sign  the 
Privacy  Act  statement  (Appendix  C).  Normally,  any  AFCT  score  is  recorded  in  personnel 
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files.  Due  to  the  uniqueness  of  the  testing  situation,  examinees  were  informed  that  their 
scores  would  only  be  changed  if  they  were  higher  than  previous  scores  as  indicated  in 
the  Privacy  Act. 

All  examinees,  regardless  of  condition,  were  given  2  pencils  and  2  pieces  of  scratch 
paper,  as  indicated  in  normal  AFCT  instructions.  Examinees  could  ask  for  additional 
pencils  or  paper  at  any  time  during  the  testing  by  raising  their  hand. 

In  the  paper-based  condition,  the  test  administrators  read  instructions  at  the 
beginning  of  each  subtest,  including  the  sample  problem(s),  the  number  of  questions  in 
that  subtest,  and  the  time  limit.  Once  all  questions  had  been  answered,  the  test 
administrators  would  start  the  subtest  and  then  post  the  start  and  end  time  on  a  board 
at  the  front  of  the  room.  Those  in  the  web-based  condition  were  not  read  that 
information  because  of  the  asynchronous  nature  of  the  testing,  but  that  information  was 
available  for  them  to  read  prior  to  starting  the  subtest. 

Data  Analyses 

Tests  were  scored  using  the  item  response  theory  methodology  laid  out  by  Segall 
(2005),  specifically  the  three  parameter  logistic  model.  Responses  were  recoded  to  be  1 
if  correct  and  o  if  incorrect  or  blank.  Theta  scores  were  computed  from  the  given 
parameters  using  the  bisection  algorithm  and  then  theta  scores  were  translated  to 
standard  scores  based  on  the  1997  ASVAB  normative  scale.  Appendix  D  contains  the 
SAS  programming^  used  for  scoring  and  converting  the  scores  to  standard  scale  scores. 

Once  scale  scores  were  computed  for  all  subjects,  an  equating  study  was  conducted. 
Because  of  the  relatively  small  number  of  cases,  linear  equating  was  chosen  (Stoloff, 
1986).  In  linear  equating,  lines  are  computed  using  the  means  (p)  and  standard 
deviations  (o)  of  both  tests  as  follows,  where  p  =  paper  and  w  =  web  (Kolen  &  Brennan, 

1995): 

p  =  (ap/aw)*w  +  (pp-((ap/aw)*Pw)) 
w  =  (aw/ap)*p  +  (pw-((aw/op)*pp)) 

Using  these  formulae,  scores  were  calculated  for  all  possible  scores  between  20  and 
80,  and  the  two  lines  were  graphed.  Lines  that  are  close  together,  especially  at  the  scale 
mean  of  50,  indicate  that  the  scores  for  the  two  tests  are  similar. 

While  standard  scores  should  range  from  20  to  80  and  are  graphed  as  such,  scores 
for  each  subtest  vary.  Table  3  shows  the  actual  range  of  standard  scores  by  subtest;  for 
the  linear  equating  graphs  that  follow,  all  subtest  scores  are  graphed  assuming  a  range 
of  20  to  80.  Since  Form  19G  was  not  constructed  using  Item  Response  Theory  and  it  was 
not  based  on  the  1997  normative  population,  the  test  score  ranges  are  inconsistent.  In 
particular,  there  are  insufficient  numbers  of  difficult  items  (all  but  one  subtest  has  a 
maximum  score  below  80)  and  in  several  cases  there  are  items  that  are  so  easy  that 
(when  missed)  produce  a  minimum  score  lower  than  20. 


^  The  author  would  like  to  thank  Dr.  David  Alderton  for  his  SAS  programming  (Appendix  D)  to  create 
scale  scores. 
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Table  3 

Standard  Score  Ranges 


Subject 

With  all 
wrong 

answers 

selected 

With  all 
correct 

answers 

selected 

1 

General  Science  (GS) 

19 

72 

2 

Arithmetic  Reasoning  (AR) 

24 

68 

3 

Word  Knowledge  (WK) 

15 

66 

4 

Paragraph  Comprehension  (PC) 

25 

62 

5 

Mathematics  Knowledge  (MK) 

27 

67 

6 

Electronics  Information  (El) 

23 

80 

7 

Automotive  and  Shop  (AS) 

23 

76 

8 

Mechanical  Comprehension  (MC) 

28 

75 

9 

Assembling  Objects  (AO) 

27 

66 

Verbal  (VE)  (Composite  of  PC  and  WK) 

16 

66 

Results 


Results  are  presented  for  the  overall  group,  male  and  female,  and  majority  and 
minority  ethnicity.  The  results  briefing  is  available  in  Appendix  E,  and  detailed  subtest 
information  by  question  is  available  in  Appendix  F.3 

General  Science 

Figure  3  shows  the  results  for  all  examinees  on  the  General  Science  (GS)  subtest.  To 
determine  the  equivalent  paper-based  score  from  the  score  obtained  when  taking  the 
web  version,  the  red  diagonal  line  (p  =  1.06W-2.41)  would  be  used  since  the  paper  score 
is  the  unknown.  The  red  arrows  show  that  a  score  of  75  on  the  web  would  be  equivalent 
to  a  score  of  77  on  paper;  paper-based  scores  are  slightly  higher  than  web-based  scores 
at  the  higher  end  of  the  score  range,  although  not  significantly  so. 


3  After  completion  of  all  analyses,  respondent  scores  were  changed  if  they  did  better  on  all  9  suhtests. 
Scores  were  changed  for  only  5  participants. 
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Figure  3.  Linear  equating  for  General  Science. 


To  convert  from  a  paper  score  to  an  equivalent  web  score,  the  blue  diagonal  line 
would  be  used.  As  indicated  by  the  blue  arrows,  a  paper  score  of  40  would  be  equivalent 
to  a  web  score  of  40.  The  dotted  black  line  is  the  identity  line,  where  the  two  versions 
would  be  considered  equivalent;  because  the  red  line  and  blue  line  overlap  with  the 
identity  line  from  33  to  50,  scores  in  that  range  are  equivalent  between  the  two  versions 
of  the  subtest.  Scores  below  33  tended  to  be  higher  for  the  web-based  subtest  and  scores 
over  50  higher  for  the  paper-based  subtest. 

Figures  4  and  5  show  the  results  for  GS  by  gender  and  ethnicity.  Scores  for  males  are 
equivalent  between  the  two  subtests  around  the  mean  of  50  (from  scores  of  45  to  scores 
of  55),  but  vary  by  3  points  at  the  extremes  in  the  same  direction  as  seen  for  all 
examinees  above.  Women  tend  to  score  significantly  higher  on  the  paper  version  of  the 
subtest  than  the  web  version,  especially  when  the  score  is  low.  However,  the  results  for 
females  are  inherently  unstable  because  of  the  small  sample  sizes,  particularly  for 
equating  purposes. 
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Male 


Female 


N 

Mean 

sd 

Paper (p) 

525 

54.62 

8.010 

Web  (w) 

552 

54.22 

7.309 

N 

Mean 

sd 

Paper (p) 

90 

53.03 

7.257 

Web  (w)  * 

111 

49.92 

7.760 

*  Significant  difference  between  paper  and  web  for  females  (f=2.908,  p=.00). 


Figure  4.  Linear  equating  for  General  Science  by  gender. 


Majority 


Actual  Score 


-p=1.07w-3.29 
-  W=.94p+3.Q9 


N 

Mean 

sd 

Paper (p) 

399 

55.66 

7.469 

Web  (w) 

353 

55.24 

6.999 

Minority 


Actual  Score 

-p=l.llw-3.93 
-  w=.90p+3.55 


N 

Mean 

sd 

Paper (p) 

166 

52.06 

8.227 

Web  (w) 

219 

50.51 

7.421 

Figure  5.  Linear  equating  for  General  Science  by  ethnicity. 

The  results  for  the  majority  ethnicity  are  similar  to  those  found  for  the  males,  with 
scores  being  equivalent  around  the  mean  (between  42  and  56  inclusive),  higher  for 
paper  for  scores  above  the  mean,  and  higher  for  web  for  scores  below  the  mean.  Scores 
for  minority  again  are  equivalent,  but  in  a  range  below  the  mean  (32  to  40).  As  with 
majority,  paper-based  scores  tend  to  be  higher  than  web-based  scores  for  the  high  end 
of  the  score  range,  but  the  difference  is  slightly  larger. 
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One-way  Analyses  of  Variance  (ANOVAs)  were  computed  for  the  scale  scores.  Table 
4  shows  that  the  only  significant  difference  in  mean  GS  scores  was  for  females. 


Table  4 

One-way  ANOVAs  for  General  Science  scale  scores 


Between 

Groups 

Sum  of 
Squares 

df 

Mean 

Square 

F 

Sig. 

Overall 

177.306 

1 

177.306 

2.919 

.088 

Males 

43.803 

1 

43.803 

.747 

.388 

Females 

482.083 

1 

482.083 

8.481 

.004 

Majority 

32.338 

1 

32.338 

.615 

.433 

Minority 

227.852 

1 

227.852 

3.766 

.053 

Arithmetic  Reasoning 

Results  for  the  Arithmetic  Reasoning  (AR)  subtest  for  all  respondents  found  the  two 
versions  of  the  subtest  to  be  equivalent  (Figure  6).  Results  for  males  show  that  the  two 
subtests  are  generally  equivalent,  with  a  one  point  difference  for  scores  below  35  (Figure 
7),  while  the  results  for  females  as  well  as  for  both  the  majority  and  minority  group 
(Figure  8)  show  more  differentiation,  especially  for  the  female  and  minority  groups.  For 
females,  paper-based  scores  tend  to  be  at  most  four  points  higher  (although  equivalent 
for  scores  69  and  above).  Results  for  majority  are  equivalent  from  50-70  and  at  most  2 
points  different  for  low  scores  while  the  results  for  minority  show  that  web-based  scores 
are  higher  by  5  points  at  the  lower  end  of  the  scale  and  paper-based  scores  are  higher  by 
2  points  at  the  higher  end  of  the  scale  (equivalent  between  56  and  65). 


Actual  Score 


N 

Mean 

sd 

Paper (p) 

660 

51.77 

7.741 

Web  (w) 

739 

51.81 

7.640 

Figure  6.  Linear  equating  for  Arithmetic  Reasoning. 
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Male 


Female 


20  25  30  35  40  45  50  55 

Actual  Score 


65  70  75 


-p  =  1.02w-1.12 
w  =  .98p+1.10 


N 

Mean 

sd 

Paper (p) 

521 

52.20 

7.788 

Web  (w) 

549 

52.37 

7.650 

N 

Mean 

sd 

Paper (p) 

90 

51.67 

6.722 

Web  (w) 

111 

49.88 

7.230 

Figure  7.  Linear  equating  for  Arithmetic  Reasoning  by  gender. 


Majority 

Minority 

UJ 

,U 

1  50  - 

% 

5  ^5 

S3 

-  ■  '.■■-''y 

.•'5^ 

30 

• 

20  25  30  35 

0  45  50  55  6 

Actual  Score 

65  70  75  80 

2 

0  25  30  35 

40  45  50  55  6 

Actual  Score 

65  70  75  8 

p  =  .95w  +  2.76 
w  =  1.05p-2.89 

*p=1.10w-6.12 
w  =  .91p+5.56 

N 

Mean 

sd 

Paper (p) 

398 

52.95 

7.420 

Web  (w) 

351 

52.61 

7.776 

N 

Mean 

Sd 

Paper (p) 

163 

50.18 

8.221 

Web  (w) 

218 

51.10 

7.461 

Figure  8.  Linear  equating  for  Arithmetic  Reasoning  by  ethnicity  group. 

Table  5  shows  the  results  of  one-way  ANOVAs;  no  significance  was  found  for  any 
group. 
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Table  5 

One-way  ANOVAs  for  Arithmetic  Reasoning  scale  scores 


Between 

Groups 

Sum  of 
Squares 

df 

Mean 

Square 

F 

Sig. 

Overall 

.672 

1 

.672 

.011 

.915 

Males 

8.262 

1 

8.262 

.139 

.710 

Females 

158.144 

1 

158.144 

3.221 

.074 

Majority 

22.254 

1 

22.254 

.386 

.534 

Minority 

78.668 

1 

78.668 

1.295 

.256 

Word  Knowledge 

The  linear  equating  for  Word  Knowledge  (WK)  showed  minimal  variance.  For  all 
examinees,  the  two  subtests  were  equivalent  for  scores  from  40  to  54.  For  higher  scores, 
the  paper  version  of  the  text  tended  to  be  higher  while  the  web  version  was  higher  when 
scores  were  below  40  (see  Figure  9). 


Figure  9.  Linear  equating  for  Word  Knowledge. 

Linear  equating  by  subgroup  again  shows  minimal  variance  for  males  but  a  larger 
difference  for  females  (See  Figure  10).  While  the  lines  for  males  cross  and  the  scores  are 
equivalent  from  46  to  57,  the  scores  for  females  are  always  higher  in  the  paper  version. 
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Male 


Female 


-  p=1.08w-4.15 
w=.93p+3.84 


N 

Mean 

sd 

Paper (p) 

515 

54.08 

7.784 

Web  (w) 

551 

53.88 

7.202 

N 

Mean 

Sd 

Paper (p) 

87 

53.94 

6.900 

Web  (w)  * 

111 

51.32 

6.762 

*  Significant  difference  between  paper  and  web  for  femaies  (?^2.682.  /7=.01). 


Figure  10.  Linear  equating  for  Word  Knowledge  by  gender. 


As  with  males,  the  majority  subgroup  shows  that  the  lines  again  cross  with  the 
subtests  being  equivalent  in  the  middle  range  of  scores  (51  to  57,  see  Figure  11).  The 
scores  for  the  minority  show  a  small  difference,  with  paper-based  scores  always  being 
higher  than  web-based  scores. 


N 

Mean 

sd 

Paper (p) 

394 

55.15 

7.289 

Web  (w) 

352 

55.06 

6.362 

Minority 


Actual  Score 

-  p=1.03w-.03 
w=.97p+.03 


N 

Mean 

Sd 

Paper (p) 

159 

52.01 

8.183 

Web  (w) 

219 

50.49 

7.940 

Figure  11.  Linear  equating  for  Word  Knowledge  by  ethnicity  group. 


One-way  ANOVAs  were  computed  for  all  groups.  Findings  are  significant  for  the 
female  sub-group,  as  indicated  in  Table  6. 
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Table  6 

One-way  ANOVAs  for  Word  Knowledge  scale  scores 


Between 

Groups 

Sum  of 
Squares 

df 

Mean 

Square 

F 

Sig. 

Overall 

65.548 

1 

65.548 

1.208 

.272 

Males 

11.197 

1 

11.197 

.200 

.655 

Females 

336.642 

1 

336.642 

7.231 

.008 

Majority 

1.694 

1 

1.694 

.036 

.850 

Minority 

212.190 

1 

212.190 

3.280 

.071 

Paragraph  Comprehension 

Paragraph  comprehension  (PC)  scores  tend  to  be  higher  for  the  paper  version  than 
the  web,  although  they  are  equivalent  for  scores  of  72  and  above  (see  Figure  12). 
Comparisons  by  subgroup  show  this  same  basic  trend,  with  lines  for  the  female 
subgroup  showing  the  largest  point  difference  (see  Figures  13  and  14). 


PC  scores  tend  to 
differ  at  the  low  end 
of  the  scale,  with 
paper  scores  higher 
than  web  scores. 


p=.91w+6.98 

w=1.10p-7.68 


N 

Mean 

sd 

Paper (p) 

647 

53.13 

7.004 

Web  (w)  + 

740 

50.79 

7.709 

Figure  12.  Linear  equating  for  Paragraph  Comprehension. 
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Male 


Female 


20  25  30  35  40  45  50  55  60  65  70  75  80 

_ Actual  Score _ 

p=.91w+6.91 
-  w=1.10p-7.60 


N 

Mean 

sd 

Paper (p) 

88 

54.53 

5.628 

Web  (w)  * 

no 

50.03 

7.343 

N 

Mean 

sd 

Paper (p) 

511 

53.30 

7.027 

Web  (w)  * 

551 

51.07 

7.736 

*  Significant  difference  between  paper  and  web  for  males  (/^4.905,  p=.00)  and  females  (/^4.741,  p=.00). 


Figure  13.  Linear  equating  for  Paragraph  Comprehension  by  gender. 


Minority 

Large  difference  for 
both  majority  and 
minority  on  PC. 


N 

Mean 

sd 

Paper (p) 

391 

54.04 

6.817 

Web  (w)  * 

352 

51.74 

7.539 

N 

Mean 

sd 

Paper (p) 

158 

52.39 

7.127 

Web  (w)  * 

218 

49.04 

7.960 

*  Significant  difference  between  paper  and  web  for  majority  (f^4.367,  yO=.00)  and  minority  (f^4.207.  /7=.00). 


Figure  14.  Linear  equating  for  Paragraph  Comprehension  by  ethnicity  group. 

Results  for  the  one-way  ANOVAs  for  the  Paragraph  Comprehension  subtest  are 
significant  for  all  groups  (see  Table  7). 
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Table  7 

One-way  ANOVAs  for  Paragraph  Comprehension  scale  scores 


Between 

Groups 

Sum  of 
Squares 

df 

Mean 

Square 

F 

Sig. 

Overall 

1886.799 

1 

1886.799 

34.561 

.000 

Males 

1316.648 

1 

1316.648 

24.021 

.000 

Females 

993.002 

1 

993.002 

22.545 

.000 

Majority 

979.919 

1 

979.919 

19.073 

.000 

Minority 

1028.741 

1 

1028.741 

17.711 

.000 

Verbal  Composite 

The  Verbal  Composite  (VE)  is  based  on  the  scale  scores  of  combined  WK  and  PC 
scores.  While  there  were  large  differences  between  lines  for  PC,  the  differences  are 
somewhat  tempered  by  the  similarity  of  WK,  so  that  the  VE  scores  are  very  similar 
between  the  two  administration  modes.  As  seen  in  Eigure  15,  the  lower  range  of  scores 
(33  and  below)  are  equivalent  between  the  two  subtests  before  the  lines  diverge  and  the 
scores  on  the  paper  version  are  slightly  higher  than  on  the  web. 


VE  scores  are  based 
on  WKand  PC 
scores. 


-  p=1.04w-.72 
w  =  .96p+0.69 


N 

Mean 

sd 

Paper (p) 

646 

53.82 

7.270 

Web  (w)  * 

739 

52.61 

7.013 

Figure  15.  Linear  equating  for  Verbal  composite. 


Lines  for  the  male  subgroup  are  similar  to  the  overall  findings,  with  equivalence  at 
the  lower  end  of  the  scale  (between  24  and  43,  inclusive)  and  the  paper  being  higher 
than  the  web  on  the  higher  end  of  the  scale  (Eigure  16).  The  scores  for  females  are  never 
equivalent,  as  seen  in  Eigure  16;  paper-based  scores  are  always  higher,  and  with  a  6 
point  difference  at  the  low  end  of  the  range  of  scores. 
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Male 


Female 


*p=1.05w-1.73 

w=.95p+1.64 


Actual  Score 


p=.92w  +  7.66 
w=1.09p-8.34 


N 

Mean 

sd 

Paper (p) 

511 

54.08 

7.399 

Web  (w)  * 

550 

53.06 

7.035 

N 

Mean 

sd 

Paper (p) 

87 

54.52 

6.344 

Web  (w)  * 

no 

50.97 

6.902 

*  Significant  difference  between  paper  and  web  for  males  (f^2.302.  p=.d2)  and  females  (f^3.714,  p=.00). 


Figure  16.  Linear  equating  for  Verbal  composite  by  gender. 


The  findings  for  the  majority  subgroup  show  equivalence  around  the  mean  (40-49). 
Lower  than  this,  scores  for  web  are  slightly  higher  while  the  scores  at  the  higher  end  of 
the  range  are  higher  for  paper  (see  Figure  17).  The  findings  for  the  minority  subgroup 
are  closer  to  those  seen  for  females;  paper-based  scores  are  always  higher,  by  as  much  as 
four  points. 


N 

Mean 

sd 

Paper (p) 

391 

55.07 

7.039 

Web  (w) 

352 

54.13 

6.386 

*  Significant  difference  between  paper  and  web  for  minority  (f^3.i 


Minority 


N 

Mean 

Sd 

Paper (p) 

158 

52.39 

7.605 

Web  (w)  * 

218 

49.93 

7.682 

5,  p=.00). 


Figure  17.  Linear  equating  for  Verbal  Composite  by  ethnicity  group. 


One-way  ANOVAs  show  significance  for  all  subgroups  except  Majority  (Table  8). 
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Table  8 

One-way  ANOVAs  for  Verbal  Composite  scale  scores 


Between 

Groups 

Sum  of 
Squares 

df 

Mean 

Square 

F 

Sig. 

Overall 

503.655 

1 

503.655 

9.896 

.002 

Males 

274.813 

1 

274.813 

5.283 

.022 

Females 

610.322 

1 

610.322 

13.751 

.000 

Majority 

166.885 

1 

166.885 

3.677 

.056 

Minority 

552.061 

1 

552.061 

9.433 

.002 

Mathematics  Knowledge 

Linear  equating  of  the  Mathematics  Knowledge  (MK)  scores  shows  that  the  two 
versions  of  the  subtest  are  similar.  The  two  are  equivalent  when  the  scores  range  from 
35  to  51;  web-based  scores  are  higher  below  that  range  and  paper-based  scores  are 
higher  above  that  range  (Figure  18).  Results  for  males  (Figure  19)  and  majority  (Figure 
20)  show  the  same  pattern,  with  scores  of  males  being  equivalent  from  37  to  52  and 
scores  for  majority  being  equivalent  from  26  to  47.  Scores  for  females  (Figure  19)  also 
show  the  same  pattern  (equivalent  from  31-40,  inclusive),  although  the  difference  in 
scores  between  the  two  versions  is  larger  than  for  males  or  overall.  The  minority 
subgroup  (Figure  20)  also  shows  the  same  general  pattern,  although  the  range  of 
equivalency  (44  to  52)  is  smaller  than  the  overall  results  and  the  differences  between 
scores  at  the  extremes  are  larger. 


N 

Mean 

sd 

Paper (p) 

648 

53.69 

7.051 

Web  (w) 

739 

53.09 

6.652 

Figure  18.  Linear  equating  for  Mathematics  Knowledge. 
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Male 


Female 


-p=1.10w-3.67 

w=.91p+3.32 


N 

Mean 

sd 

Paper (p) 

512 

54.05 

7.101 

Web  (w) 

549 

53.50 

6.695 

N 

Mean 

sd 

Paper (p) 

88 

53.61 

6.510 

Web  (w) 

111 

51.94 

5.903 

Figure  19.  Linear  equating  for  Mathematics  Knowledge  by  gender. 


Majority 


Minority 


N 

Mean 

sd 

Paper (p) 

392 

54.60 

7.003 

Web  (w) 

350 

53.83 

6.705 

N 

Mean 

Sd 

Paper (p) 

158 

52.50 

7.173 

Web  (w) 

219 

52.02 

6.427 

Figure  20.  Linear  equating  for  Mathematics  Knowledge  by  ethnicity  group. 


One-way  ANOVAs  show  no  significant  differences  for  the  groups  (Table  9). 
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Table  9 

One-way  ANOVAs  for  Mathematics  Knowledge  scale  scores 


Between 

Groups 

Sum  of 
Squares 

df 

Mean 

Square 

F 

Sig. 

Overall 

128.131 

1 

128.131 

2.737 

.098 

Males 

79.007 

1 

79.007 

1.662 

.198 

Females 

137.995 

1 

137.995 

3.615 

.059 

Majority 

108.269 

1 

108.269 

2.298 

.130 

Minority 

21.300 

1 

21.300 

.468 

.495 

Electronics  I  nformation 

The  scores  for  the  Electronics  Information  (El)  subtest  show  similarity  between  the 
two  groups;  almost  consistently  through  the  range  of  scores  there  is  a  one  point 
difference  between  the  paper-based  and  web-based  scores  (Eigure  21).  Males  (Eigure 
22)  and  the  majority  subgroup  (Eigure  23)  show  this  same  tendency,  with  males  scoring 
exactly  the  same  on  the  two  versions  except  at  the  maximum  (77  and  above)  and 
majority  being  exactly  the  same  above  a  score  of  47  and  one  point  different  below  that. 


N 

Mean 

sd 

Paper (p) 

647 

57.79 

11.675 

Web  (w) 

735 

56.70 

11.839 

Figure  21.  Linear  equating  for  Electronics  I  nformation. 

Noticeable  differences  were  found  for  the  scores  of  females  (Eigure  22)  and  minority 
(Eigure  23).  Scores  on  the  web-based  version  are  5-7  points  lower  for  females  than  on 
the  paper-based  version;  the  difference  ranges  from  3-4  points  for  minority. 
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Male 


Female 


-^p=1.00w+.01 
w  =  .99p-.01 


N 

Mean 

sd 

Paper (p) 

511 

59.22 

11.709 

Web  (w) 

545 

58.83 

11.634 

20  25  30  35  40  45  50  55  60  65  70 

_ Actual  Score _ 


-^p  =  1.05w  +  2.59 
w  =  .95p-2.46 


N 

Mean 

sd 

Paper (p) 

88 

52.90 

9.047 

Web  (w)  * 

111 

47.77 

8.590 

*  Significant  difference  between  paper  and  web  for  females  (f^4.087,  p=.00). 


Figure  22.  Linear  equating  for  Electronics  I  nformation  by  gender. 


Majority 


Minority 


p  =  .98w+1.66 
w  =  1.03p-1.70 


N 

Mean 

sd 

Paper (p) 

391 

59.71 

11.311 

Web  (w) 

347 

59.53 

11.600 

N 

Mean 

sd 

Paper (p) 

158 

55.33 

11.601 

Web  (w)  * 

218 

52.61 

11.513 

*  Significant  difference  between  paper  and  web  for  minority  (t^2.254,  p=.03). 


Figure  23.  Linear  equating  for  Electronics  I  nformation  by  ethnicity  group. 


One-way  ANOVAs  show  a  significant  difference  between  paper  and  web  for  only  the 
females  (Table  lo). 
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Table  10 

One-way  ANOVAs  for  Electronics  I  nformation  scale  scores 


Between 

Groups 

Sum  of 
Squares 

df 

Mean 

Square 

F 

Sig. 

Overall 

408.580 

1 

408.580 

2.953 

.086 

Males 

39.724 

1 

39.724 

.292 

.589 

Females 

1292.764 

1 

1292.764 

16.715 

.000 

Majority 

5.837 

1 

5.837 

.045 

.833 

Minority 

679.541 

1 

679.541 

5.094 

.025 

Automotive  and  Shop 

Linear  equating  for  Automotive  and  Shop  (AS)  scores  shows  that  paper  versions  of 
the  subtest  tend  to  score  i  or  2  points  higher  for  those  who  score  61  or  below  (Figure 
24).  For  those  scores  at  the  high  end  of  the  range,  the  two  subtests  are  equivalent.  This 
general  pattern  holds  true  for  males  (Figure  25),  although  the  subtest  is  even  more 
equivalent  (from  scores  29  and  up). 


N 

Mean 

sd 

Paper (p) 

646 

51.61 

9.854 

Web  (w) 

731 

50.73 

10.227 

Figure  24.  Linear  equating  for  Automotive  and  Shop. 


23 


Male 


Female 


N 

Mean 

sd 

Paper (p) 

88 

44.94 

6.464 

Web  (w)  * 

no 

42.19 

6.128 

N 

Mean 

sd 

Paper (p) 

510 

53.01 

9.803 

Web  (w) 

543 

52.73 

9.890 

*  Significant  difference  between  paper  and  web  for  females  (^^3.062,  p=.00). 


Figure  25.  Linear  equating  for  Automotive  and  Shop  by  gender. 


Scores  for  females  show  divergence  between  paper-based  and  web-based,  with 
paper-based  scores  tending  to  be  four  points  higher  or  more  (Figure  25).  The  difference 
between  the  subtests  increases  as  the  scores  increase. 


Graphs  for  both  majority  and  minority  show  areas  of  equivalence  (Figure  26).  The 
subtests  are  generally  equivalent  for  the  majority  subgroup,  with  a  1  point  difference 
(paper  higher)  for  those  below  32  and  a  1  point  difference  (web  higher)  for  scores  above 
63.  The  reverse  trend  is  true  for  minority;  web-based  scores  are  higher  below  27  and 
paper-based  scores  are  higher  above  45. 


Majority 


Minority 


N 

Mean 

sd 

Paper (p) 

390 

53.95 

9.401 

Web  (w) 

346 

54.15 

9.692 

N 

Mean 

Sd 

Paper (p) 

158 

46.68 

9.216 

Web  (w) 

217 

46.16 

8.769 

Figure  26.  Linear  equating  for  Automotive  and  Shop  by  ethnicity  group. 
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One-way  ANOVAs  show  the  similarity  between  the  two  versions  of  the  subtest  for  all 
groups  except  females,  where  the  subtest  mode  causes  significant  differences  in  the 
scores  (Table  ii). 


Table  11 

One-way  ANOVAs  for  Automotive  and  Shop  scale  scores 


Between 

Groups 

Sum  of 
Squares 

df 

Mean 

Square 

F 

Sig. 

Overall 

270.513 

1 

270.513 

2.676 

.102 

Males 

20.950 

1 

20.950 

.216 

.642 

Females 

370.334 

1 

370.334 

9.393 

.002 

Majority 

7.473 

1 

7.473 

.082 

.774 

Minority 

24.773 

1 

24.773 

.309 

.579 

Mechanical  Comprehension 

Scores  for  the  Mechanical  Comprehension  (MC)  are  consistently  different;  with 
paper-based  scores  being  two  points  higher  than  web-based  scores  (see  Figure  27).  This 
trend  is  generally  consistent  for  each  of  the  subgroups  at  the  higher  end  of  the  score 
range  (see  Figures  28  and  29).  For  lower  scores,  differences  between  the  two  subtests 
are  smaller  (equivalent  for  majority  for  scores  33  and  lower). 


Actual  Score 


Consistent  difference 
between  paper  and 
web  for  MC,  with 
paper  being  higher. 


p=1.01w+1.34 
I  w=.99p-1.32 


N 

Mean 

sd 

Paper (p) 

646 

57.34 

8.932 

Web  (w)  » 

726 

55.32 

8.824 

*  Significant  difference  between  paper  and  web  (f^4.208,  p=.00). 

Figure  27.  Linear  equating  for  Mechanical  Comprehension. 
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Male 


Female 


N 

Mean 

sd 

Paper (p) 

510 

58.28 

8.912 

Web  (w)  * 

539 

56.47 

8.786 

*  Significant  difference  between  paper  and  web  for  males  (f^3.312, 


N 

Mean 

sd 

Paper (p) 

88 

53.98 

7.680 

Web  (w)  * 

109 

49.95 

7.033 

p=.00)  and  females  (f^3.837,  p=.00). 


Figure  28.  Linear  equating  for  Mechanical  Comprehension  by  gender. 


Majority 


-p=1.03w-.77 

w=.96p+.75 


Minority 


N 

Mean 

sd 

Paper (p) 

390 

58.92 

8.725 

Web  (w) 

345 

57.51 

8.405 

N 

Mean 

sd 

Paper (p) 

158 

54.66 

8.688 

Web  (w)  * 

215 

51.73 

8.569 

*  Significant  difference  between  paper  and  web  for  minority  (/^3.244,  /7=.00). 


Figure  29.  Linear  equating  for  Mechanical  Comprehension  by  ethnicity  group. 


As  shown  in  Table  12,  results  of  one-way  ANOVAs  indicate  significant  differences 
between  the  two  subtests’  means  for  all  groups  except  majority. 
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Table  12 

One-way  ANOVAs  for  Mechanical  Comprehension  scale 

scores 


Between 

Sum  of 

Mean 

Groups 

Squares 

df 

Square 

F 

Sig. 

Overall 

1385.993 

1 

1385.993 

17.597 

.000 

Males 

859.454 

1 

859.454 

10.979 

.001 

Females 

788.087 

1 

788.087 

14.671 

.000 

Majority 

365.632 

1 

365.632 

4.971 

.026 

Minority 

781.673 

1 

781.673 

10.521 

.001 

Assembling  Objects 

Large  differences  between  the  two  versions  of  the  subtest  were  found  for  all 
subgroups  (see  Figures  30,  31,  and  32).  Differences  are  largest  for  lower  scores  while 
overall,  males,  and  majority  analyses  show  scores  only  differ  by  one  point  for  higher 
scores. 


Actual  Score 

*  Significant  difference  between  paper  and  web  (/^6.786,  p=.00). 


Large  difference 
between  AO  scores 
at  the  low  end  of  the 
scale. 


p=.91w+8.43 

w=1.10p-9.30 


N 

Mean 

sd 

Paper (p) 

644 

55.10 

9.320 

Web  (w)  * 

721 

51.48 

10.281 

Figure  30.  Linear  equating  for  Assembling  Objects. 


27 


Female 


Male 


N 

Mean 

sd 

Paper (p) 

87 

56.29 

8.215 

Web  (w)  * 

109 

49.97 

9.448 

N 

Mean 

sd 

Paper (p) 

510 

55.05 

9.529 

Web  (w)  * 

534 

52.02 

10.310 

*  Significant  difference  between  paper  and  web  for  maies  (f^4.925,  p=.00)  and  femaies  (f^4.927,  p=.00). 


Figure  31.  Linear  equating  for  Assembling  Objects  by  gender. 


Majority 


Minority 


N 

Mean 

sd 

Paper (p) 

390 

55.87 

9.271 

Web  (w)  * 

343 

52.15 

10.225 

N 

Mean 

sd 

Paper (p) 

158 

53.77 

9.860 

Web  (w)  * 

213 

50.28 

10.437 

*  Significant  difference  between  paper  and  web  for  majority  (f=5.165,  /3=.00)  and  minority  (f=3.260,  p=.00). 


Figure  32.  Linear  equating  for  Assembling  Objects  by  ethnicity  group. 

Differences  between  the  two  subtests’  means  are  significant  for  all  groups,  as 
indicated  in  Table  13,  with  the  paper  version  yielding  higher  scores  than  the  web 
version. 


28 


Table  13 

One-way  ANOVAs  for  Assembling  Objects  scale  scores 


Between 

Groups 

Sum  of 
Squares 

df 

Mean 

Square 

F 

Sig. 

Overall 

4452.975 

1 

4452.975 

45.999 

.000 

Males 

2398.373 

1 

2398.373 

24.294 

.000 

Females 

1929.389 

1 

1929.389 

24.235 

.000 

Majority 

2522.717 

1 

2522.717 

26.651 

.000 

Minority 

1101.164 

1 

1101.164 

10.594 

.001 

Armed  Forces  Qualification  Test 

The  Armed  Forces  Qualification  Test  (AFQT)  is  a  composite  of  three  scores  from  the 
ASVAB:  AR,  MK,  and  two  times  the  VE  score.  Although  not  generally  computed  while  in 
the  service  and  therefore  not  a  product  of  the  AFCT,  scores  were  computed  following 
this  test  to  further  compare  the  two  versions.  As  Figure  33  shows,  the  two  versions  yield 
similar  scores  (1  point  difference  at  the  lowest  end,  increasing  to  a  3  point  difference  for 
higher  scores);  paper-based  scores  tend  to  be  slightly  higher. 


p - 

Armed  Forces 
Qualification  Test 
(AFQT)  is 

AR+MK+2VE;  scores 
range  from  1  to  99. 


-^p=1.02w-2.30 

w=.99p-2.26 


N 

Mean 

sd 

Paper (p) 

646 

61.72 

22.019 

Web  (w)  + 

734 

58.54 

21.693 

*  Significant  difference  between  paper  and  web  (f^2.695,  /j=.01). 


Figure  33.  Linear  equating  for  Armed  Forces  Qualification  Test. 


The  differences  for  males  are  similar  to  the  overall  findings  but  differences  for 
females  are  much  larger  (see  Figure  34).  Differences  in  female  scores  range  from  15  at 
the  lower  end  to  6  at  the  higher  end. 
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Male 


Female 


N 

Mean 

sd 

Paper (p) 

511 

63.06 

21.972 

Web  (w)  * 

545 

60.40 

21.343 

N 

Mean 

sd 

Paper (p) 

87 

62.62 

19.524 

Web  (w)  * 

110 

52.54 

21.787 

*  Significant  difference  between  paper  and  web  for  males  (f=1.993,  /O=.05)  and  females  (f^3.376,  p=.00). 


Figure  34.  Linear  equating  for  Armed  Forces  Qualification  Test  by  gender. 


Neither  the  majority  nor  the  minority  subgroup  show  differences  as  large  as  those 
found  for  females,  but  the  two  test  versions  are  more  dissimilar  than  was  found  for 
males  (Figure  35).  Differences  for  majority  range  from  four  points  at  the  lower  end  of 
scores  to  three  points  at  the  higher  end  while  differences  for  minority  range  from  three 
points  at  the  lower  end  to  six  points  at  the  higher  end. 


Majority 


Actual  Score 


-.-p=.99w+3.93 

w=1.01p-3.96 


N 

Mean 

sd 

Paper (p) 

391 

66.21 

20.406 

Web  (w)  * 

349 

62.88 

20.602 

*  Significant  difference  between  paper  and  web  for  majority  {^^2.; 


Minority 


Actual  Score 


-^p=1.03w+3.30 
-  w=.97p-3.21 


N 

Mean 

Sd 

Paper (p) 

158 

56.46 

23.062 

Web  (w)  * 

217 

51.76 

22.457 

),  p=.03)  minority  (f^l.977,  p=.05). 


Figure  35.  Linear  equating  for  Armed  Forces  Qualification  Test  by  ethnicity 

group. 


Significant  differences  were  found  for  all  analysis  groups,  as  indicated  in  Table  14. 
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Table  14 

One-way  ANOVAs  for  AFQT  scores 


Between 

Groups 

Sum  of 
Squares 

df 

Mean 

Square 

F 

Sig. 

Overall 

3465.925 

1 

3465.925 

7.262 

.007 

Males 

1861.638 

1 

1861.638 

3.972 

.047 

Females 

4940.142 

1 

4940.142 

11.398 

.001 

Majority 

2045.293 

1 

2045.293 

4.867 

.028 

Minority 

2015.660 

1 

2015.660 

3.907 

.049 

Discussion 


Findings 

The  equating  study  presented  here  shows  that  the  version  of  the  AFCT  as 
programmed  in  WebQuiz  XP  was  fully  functional.  The  only  issues  that  were 
encountered  related  to  connectivity  of  the  laptops;  the  room  as  configured  had  many 
cables  that  were  occasionally  and  inadvertently  pulled  out  of  the  laptops. 

While  there  are  key  benefits  to  conducting  the  AFCT  through  the  web,  the  results 
shown  here  indicate  that  there  are  differences  between  the  paper  and  web  versions  that 
need  to  be  accounted  for  before  using  a  web  version  Navy- wide.  Of  key  note,  paper- 
based  scores  tend  to  be  slightly  higher,  significantly  for  PC  (5  points  different  for  low 
scores)  and  AO  (7  points  different  for  low  scores).  This  was  true  for  the  overall  group,  as 
well  as  males  and  both  majority  and  minority  ethnicity.  The  differences  were  even 
higher  for  females  (about  10  points),  although  this  likely  is  influenced  by  the  low 
number  of  females  who  participated. 

Testing  Observations 

While  administering  the  test,  there  were  both  advantages  and  disadvantages 
observed  by  the  test  proctors.  These  advantages  and  disadvantages  should  be  considered 
by  Navy  leadership  in  determining  how  to  implement  the  web-based  AFCT  if  they  decide 
to  do  so. 

The  first  overwhelming  advantage  was  that  the  web-based  AFCT  was  self-paced. 
Because  of  this.  Sailors  generally  finished  sooner  than  they  would  have  on  paper,  where 
the  pace  is  set  by  the  slowest  test-taker.  Testing  times  for  the  individual  subtests  require 
a  minimum  of  2.5  hours  for  completion,  plus  additional  time  to  read  instructions.  As 
seen  in  the  web-based  testing  condition,  some  respondents  finished  within  an  hour, 
with  at  least  half  of  the  examinees  done  within  two  hours  (consistent  with  CAT-ASVAB 
and  results  found  when  other  tests  have  moved  away  from  lock-step  administration). 
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This  shortened  testing  time  then  can  return  at  least  some  Sailors  to  their  duties,  instead 
of  spending  their  time  waiting  for  fellow  Sailors  to  complete  the  test. 

Additionally,  as  designed  here,  there  is  less  interaction  with  a  test  administrator, 
normally  another  Sailor.  The  test  administrator  is  still  needed  to  monitor  the  test-taking 
to  ensure  there  is  no  cheating  and  to  hand  out  additional  paper  and  pencils  as 
requested,  but  he  or  she  does  not  have  to  give  any  additional  instructions.  Since  the  test 
is  self-paced,  this  Sailor  may  also  return  to  duty  sooner  than  he/she  would  be  able  to 
with  a  paper-based  test. 

Because  the  examinee  records  their  test  answers  as  they  take  the  test,  there  is  no 
need  for  additional  scanning  or  hand-entry  of  the  answers,  thereby  reducing  errors  that 
may  be  associated  with  scanners  (bubbles  not  aligned,  incomplete  fill  of  the  bubble,  2 
bubbles  inadvertently  marked  with  a  succeeding  question  left  empty,  bubble  not  dark, 
etc.)  or  hand  entry.  The  computer  also  makes  immediate  scoring  and  feedback  of  results 
a  possibility. 

If  the  test  is  on  a  computer  which  is  password  protected,  or  if  the  test  continues  to 
have  an  administrative  password,  the  test  requires  less  effort  to  ensure  security. 

Multiple  computers  may  access  the  same  computer  file  of  questions,  instead  of  the  need 
to  have  multiple  copies  of  the  test  booklet  available. 

The  web-based  test  as  designed  here  made  return  to  previously  completed  subtests 
nearly  impossible  since  the  examinee  would  have  to  restart  the  entire  test.  In  the  paper- 
based  test,  ensuring  that  examinees  do  not  return  to  previous  subtests  requires  close, 
constant  vigilance  by  the  test  administrator. 

The  disadvantages  of  the  web-based  AFCT,  as  programmed,  related  to  the  inherent 
capabilities  of  a  computer.  All  Windows-based  computers  have  a  calculator  installed; 
the  test  administrators  attempted  to  delete  the  calculator  but  it  would  return  as  a 
default.  The  calculator  was  removed  from  the  Programs  menu  for  these  tests,  making  it 
more  difficult  to  find,  but  the  calculator  program  was  still  available  and  the  test 
administrators  were  especially  vigilant  to  ensure  that  it  was  not  used  on  the  test,  as 
indicated  in  the  paper-based  instructions. 

The  web-based  AFCT  was  administered  through  the  standard  version  of  Internet 
Explorer,  which  has  several  functions  installed  by  default.  One  of  these  is  the  “Find” 
function;  in  the  Paragraph  Comprehension  subtest,  “Find”  could  be  used  to  look  for  key 
words  mentioned  in  the  actual  question. 

A  final  disadvantage  observed  in  the  testing  was  the  inability  to  draw  on  the  test 
diagrams.  While  drawing  in  the  test  booklet  is  not  allowed,  it  would  still  happen  (the  test 
booklets  were  scanned  after  each  use  to  ensure  there  were  no  pencil  marks;  all  pencil 
marks  were  erased  or  the  booklet  would  no  longer  be  used).  The  marks  were  usually 
either  directional  arrows,  used  by  the  respondent  to  figure  out  how  something  worked, 
or  marks  to  help  the  examinee  keep  track  of  which  answer  choices  were  not  possible. 
Web-based  examinees  could  draw  the  diagrams  on  their  scratch  paper,  however  this 
would  be  an  unwanted  delay  in  a  timed  test;  few  examinees  drew  the  diagrams  on  their 
scratch  paper. 
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Future  Steps 

While  there  were  significant  differences  found,  some  of  these  may  be  alleviated  by 
modifications  to  the  testing  protocol.  For  AO  in  particular,  allowing  grease  pencils  on 
the  monitor  itself  or  on  a  transparency  overlay  of  the  monitor  would  allow  respondents 
to  make  notations  similar  to  what  often  happens  in  paper  test  booklets;  grease  pencil 
markings  would  be  relatively  easy  to  wipe  clean,  similar  to  using  the  eraser  when 
marking  in  a  test  booklet. 

Differences  for  the  PC  subtest  may  be  more  difficult  to  alleviate.  Studies  have  shown 
reading  speed  may  be  equivalent  between  web  and  paper  mediums  but  that  skimming 
speed  is  slower  on  a  computer  monitor  than  on  paper  (Muter  &  Maurutto,  1991).  In  a 
test  setting,  it  is  possible  that  more  are  skimming  than  are  reading;  perhaps  increasing 
the  font  size  or  changing  the  font  type  would  increase  skimming  (and  reading)  speeds 
and  these  options  should  be  considered. 

Once  another  variation  of  the  test  is  finalized— alleviating  some  of  the  potential 
causes  of  difference— this  equating  study  would  need  to  be  replicated.  This  could  also 
work  to  increase  the  dataset  and  provide  additional  data  from  females  for  analyses, 
potentially  eliminating  many  of  the  large  differences  found  for  females. 

It  is  possible  that  no  modifications  can  be  made  to  the  web-based  version  of  the  test 
so  that  the  scores  match  exactly  what  would  have  been  received  on  the  paper-based 
version.  A  decision  then  needs  to  be  made  at  the  Department  of  Defense  level  as  to  when 
to  stop  testing  modifications  so  that  a  final  version  can  be  made  available.  Additionally, 
the  DoD-level  working  group  needs  to  determine  how  to  handle  both  paper-  and  web- 
based  scores  concurrently  in  databases,  since  it  is  unlikely  all  tests  will  be  administered 
in  a  single  format  for  the  next  several  years.  If  the  assumption  is  that  most  people  will  be 
moved  to  a  web-based  version  of  the  test,  then  it  may  be  appropriate  to  create  tables 
that  easily  convert  all  paper-based  scores  to  web-based  scores  for  recording  into 
personnel  databases. 

Once  the  decision  is  made  to  convert  to  a  predominant  web-based  environment,  each 
service  will  likely  consider  redoing  their  job  analyses;  the  Navy  may  consider  modifying 
cut-scores  to  better  reflect  the  web-based  version  of  the  test.  A  concept  of  operations 
(CONOPS)  also  needs  to  be  determined  for  the  Navy  that  maintains  test  integrity  while 
also  including  automated  scoring  with  IRT  and  logging  of  scores.  While  a  DOD  version 
would  be  ideal  (similar  to  CAT-ASVAB),  this  is  at  present  not  possible  for  ships  and 
submarines,  who  may  have  limited  or  no  internet  connectivity.  If  the  decision  is  to  go 
completely  web-based,  each  ship/submarine  would  need  to  have  a  version  loaded  on 
their  servers  that  would  populate  scores  into  personnel  files  when  connectivity  became 
available. 

While  the  web-based  version  of  the  test  does  seem  attractive  for  a  number  of  reasons 
including  lack  of  printing  costs,  potential  for  increased  security  and  accuracy,  and 
reduced  Sailor  time  (both  administering  and  taking),  these  CONOPS  issues  are  key  to 
ensuring  that  testing  continues  to  allow  for  accurate  and  fair  gauging  of  rating  success. 
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Appendix  A: 
Additional  Analyses 


A-o 


Additional  Analyses 


When  the  Manpower  Accession  Policy  Working  Group  (MAPWG)  was  formally 
briefed  on  the  study  in  January  of  2008,  they  expressed  concern  that  the  two  testing 
groups  (paper-based  and  web-based)  were  fundamentally  different,  which  led  to  the 
significant  differences  between  scores  for  the  two  versions.  They  requested  that 
additional  analyses  be  conducted  using  ASVAB  scores  of  record,  comparing  the  two 
groups  for  pre-existing  data. 

ASVAB  record  data  was  extracted  for  those  personnel  with  useable  scores.  A  few  had 
record  data  scores  of  o  (predominantly  for  the  AO  sub-test,  which  has  only  recently  been 
approved  as  a  Navy  special  test  added  to  the  ASVAB).  Any  sub-test  score  that  was  o  was 
removed  from  comparisons  for  both  the  record  scores  as  well  as  the  testing  data  so  that 
those  scores  (and  Sailors)  would  not  impact  the  computation  of  the  means  and  standard 
deviations.  T-tests  of  independent  means  were  conducted,  comparing  scores  of  the  web- 
based  group  to  the  paper-based  group. 

Overall  results 

Overall  results  show  that  there  were  a  few  significant  differences  between  the  groups 
prior  to  testing,  particularly  in  the  language  areas  (WK,  PC,  and  VE),  as  well  as  in  El 
(see  Table  A-i). 


Table  A- 1 

Record  data  from  all  respondents 


Those  who  took  web- 
based  AFCT 

Those  who  took  paper- 
based  AFCT 

T 

P 

N 

Mean 

SD 

N 

Mean 

SD 

GS 

723 

54.09 

7.431 

657 

54.85 

7.446 

-1.896 

.058 

AR 

722 

54.76 

6.303 

654 

54.90 

6.485 

-.406 

.685 

WK* 

724 

52.39 

6.995 

645 

53.43 

6.864 

-2.770 

.006 

PC* 

722 

53.81 

6.079 

642 

54.71 

5.799 

-2.789 

.005 

MK 

722 

55.93 

5.396 

644 

56.26 

5.586 

-.922 

.357 

El* 

718 

54.69 

8.637 

643 

56.03 

8.386 

-2.897 

.004 

AS 

714 

51.88 

9.115 

642 

52.99 

8.511 

-2.310 

.021 

MC 

707 

55.67 

8.364 

642 

56.49 

8.078 

-1.828 

.068 

AO 

629 

57.17 

6.974 

584 

57.63 

7.318 

-1.121 

.263 

VE* 

722 

53.09 

6.295 

641 

54.11 

6.103 

-3.029 

.003 

AFQT 

716 

64.41 

18.065 

642 

66.83 

17.778 

-2.483 

.013 

A-i 


The  testing  data  was  also  compared  for  those  personnel  with  sufficient  record  data. 
As  indicated  in  Table  A-2,  there  continue  to  be  significant  differences  in  PC  and  VE, 
perhaps  caused  by  the  pre-existing  differences  discussed  above.  However,  MC  and  AO 
now  show  significant  differences;  considering  personnel  were  similar  on  MC  and  AO 
when  they  originally  took  the  ASVAB,  it  is  possible  that  the  differences  now  shown  are 
due  to  the  mode  of  administration.  Additionally,  the  pre-existing  difference  for  El  has 
disappeared,  again  suggesting  a  change  due  to  the  mode  of  administration. 


Table  A-2 

Testing  data  for  all  respondents 


Those  who  took  web- 
based  AFCT 

Those  who  took  paper- 
based  AFCT 

T 

P 

N 

Mean 

SD 

N 

Mean 

SD 

GS 

723 

53.34 

7.545 

657 

54.02 

8.015 

-1.623 

.105 

AR 

722 

51.88 

7.629 

654 

51.79 

7.739 

.217 

.828 

WK 

724 

53.42 

7.155 

645 

53.77 

7.627 

-.876 

.381 

PC* 

722 

50.87 

7.679 

642 

53.13 

7.026 

-5.646 

.000 

MK 

722 

55.93 

5.396 

644 

56.26 

5.586 

-1.513 

.130 

El 

718 

56.74 

11.868 

643 

57.77 

11.644 

-1.613 

.107 

AS 

714 

50.81 

10.231 

642 

51.58 

9.861 

-1.407 

.160 

MC* 

707 

55.37 

8.854 

642 

57.34 

8.939 

-4.063 

.000 

AO* 

629 

51.86 

10.202 

584 

55.17 

9.233 

-5.090 

.000 

VE* 

722 

52.69 

7.008 

641 

53.82 

7.290 

-2.915 

.000 

AFQT 

716 

58.85 

21.628 

642 

61.76 

22.011 

-2.454 

.014 

Gender 

Record  data  for  males  in  the  two  administration  groups  show  no  significant 
differences,  indicating  that  they  were  reasonably  similar  prior  to  the  equating  study  (see 
Table  A-3).  Eindings  for  females  show  three  differences;  WK  and  VE  are  significantly 
different  between  the  two  groups  (similar  to  overall)  as  well  as  MC  (see  Table  A-4). 


Table  A-3 

Record  data  for  males 


Those  who  took  web- 
based  AFCT 

Those  who  took  paper- 
based  AFCT 

T 

P 

N 

Mean 

SD 

N 

Mean 

SD 

GS 

548 

55.12 

6.945 

521 

55.81 

7.034 

-1.614 

.107 

AR 

547 

55.48 

6.201 

517 

55.37 

6.354 

.286 

.775 

WK 

549 

52.93 

7.005 

511 

53.80 

6.703 

-2.063 

.039 

PC 

548 

54.07 

6.121 

508 

54.89 

5.872 

-2.218 

.027 

MK 

547 

56.00 

5.494 

509 

54.46 

5.735 

-1.331 

.183 

El 

543 

56.16 

8.334 

508 

57.28 

8.078 

-2.210 

.027 

AS 

541 

53.67 

8.693 

507 

54.35 

8.172 

-1.303 

.193 

MC 

535 

57.18 

7.936 

507 

57.64 

7.912 

-.937 

.349 

AO 

473 

57.63 

6.853 

462 

57.92 

7.420 

-.633 

.527 

VE 

548 

53.54 

6.292 

508 

54.44 

5.996 

-2.376 

.018 

AFQT 

542 

65.99 

18.013 

508 

68.02 

17.645 

-1.843 

.066 

Table  A-4 

Record  data  for  females 


Those  who  took  web- 
based  AFCT 

Those  who  took  paper- 
based  AFCT 

T 

P 

N 

Mean 

SD 

N 

Mean 

SD 

GS 

110 

49.99 

7.933 

90 

52.10 

7.801 

-1.885 

.061 

AR 

110 

52.27 

5.809 

90 

53.12 

7.166 

-.927 

.355 

WK* 

110 

50.25 

6.664 

87 

53.15 

7.313 

-2.905 

.004 

PC 

109 

52.53 

5.690 

88 

54.61 

5.621 

-2.565 

.011 

MK 

110 

55.75 

5.087 

88 

556.20 

4.797 

-.634 

.527 

El 

110 

48.81 

6.798 

88 

50.02 

7.171 

-1.215 

.226 

AS 

109 

43.80 

5.434 

88 

46.06 

6.970 

-2.557 

.011 

MC* 

108 

48.83 

5.783 

88 

51.42 

6.612 

-2.924 

.004 

AO 

100 

55.56 

6.450 

79 

57.11 

6.528 

-1.588 

.114 

VE* 

109 

51.07 

6.901 

87 

54.52 

6.344 

-2.971 

.003 

AFQT 

109 

58.57 

16.727 

87 

64.95 

18.197 

-2.551 

.012 

The  data  obtained  through  the  testing  shows  three  significant  differences  for  males; 
results  for  PC,  MC,  and  AO  show  significant  differences  between  administration  modes 
where  none  had  existed  prior  to  our  testing  (see  Table  A-5). 


Table  A- 5 

Testing  data  for  males 


Those  who  took  web- 
based  AFCT 

Those  who  took  paper- 
based  AFCT 

T 

P 

N 

Mean 

SD 

N 

Mean 

SD 

GS 

548 

54.26 

7.298 

524 

54.64 

8.015 

oo 

1 

.417 

AR 

547 

52.38 

7.660 

517 

52.20 

7.786 

.380 

.704 

WK 

549 

53.89 

7.213 

511 

54.06 

7.796 

-.369 

.712 

PC* 

548 

54.10 

7.742 

508 

53.31 

7.042 

-4.840 

.000 

MK 

547 

53.51 

6.702 

509 

54.07 

7.086 

-1.320 

.187 

El 

543 

58.84 

11.653 

508 

59.19 

11.669 

-.486 

.627 

AS 

541 

52.70 

9.896 

507 

54.35 

8.172 

-.476 

.634 

MC* 

535 

56.51 

8.789 

507 

58.29 

8.912 

-3.245 

.001 

AO* 

743 

52.39 

10.246 

462 

55.18 

9.394 

-4.337 

.000 

VE 

548 

53.07 

7.045 

508 

54.08 

7.411 

-2.270 

.023 

AFQT 

542 

60.45 

21.384 

508 

63.09 

21.944 

-1.974 

.049 

There  are  a  number  of  significant  differences  between  the  two  groups  of  women 
when  viewing  the  testing  data  for  the  females.  As  mentioned  before,  the  two  groups  were 
significantly  different  for  MC  and  VE  scores  prior  to  our  testing,  and  these  differences 
continue  to  exist  (see  table  A-6)  and  the  differences  in  WK  scores  are  almost 
significantly  different.  In  addition,  GS,  PC,  El,  AS,  and  AO,  as  well  as  the  AEQT  score 
now  are  significantly  different,  possibly  indicating  a  difference  based  on  the  mode  of 
testing,  although  also  likely  impacted  by  the  small  number  of  women. 


Table  A-6 

Testing  data  for  females 


Those  who  took  web- 
based  AFCT 

Those  who  took  paper- 
based  AFCT 

T 

P 

N 

Mean 

SD 

N 

Mean 

SD 

GS* 

110 

50.00 

7.748 

90 

53.03 

7.257 

-2.831 

.005 

AR 

110 

49.96 

7.212 

90 

51.67 

6.722 

-1.720 

.087 

WK 

110 

51.38 

6.757 

87 

53.94 

6.900 

-2.616 

.010 

PC* 

109 

50.06 

7.366 

88 

54.53 

5.628 

-4.693 

.000 

MK 

110 

51.95 

5.927 

88 

53.61 

6.510 

-1.874 

.062 

El* 

110 

47.84 

8.596 

88 

52.90 

9.047 

-4.021 

.000 

AS* 

109 

42.22 

6.148 

88 

44.94 

6.464 

-3.017 

.003 

MC* 

108 

49.98 

7.060 

88 

53.98 

7.680 

-3.792 

.000 

AO* 

100 

49.99 

9.615 

79 

56.37 

8.426 

-4.652 

.000 

VE* 

109 

51.04 

6.901 

87 

54.52 

6.344 

-3.635 

.000 

AFQT^ 

109 

52.77 

21.748 

87 

62.62 

19.524 

-3.295 

.001 

Ethnicity 


As  with  males,  no  significant  differences  were  found  for  record  data  of  the  majority 
group;  the  web-based  majority  and  the  paper-based  majority  had  similar  scores  prior  to 
our  testing  (see  Table  A-y).  The  two  groups  of  minority  personnel  were  significantly 
different  on  WK,  PC,  MK,  and  VE  prior  to  testing  (see  Table  A-8). 


Table  A-7 

Record  data  for  majority 


Those  who  took  web- 
based  AFCT 

Those  who  took  paper- 
based  AFCT 

T 

P 

N 

Mean 

SD 

N 

Mean 

SD 

GS 

352 

56.03 

6.361 

395 

56.65 

6.269 

-1.340 

.181 

AR 

350 

55.92 

6.019 

394 

55.90 

6.358 

.044 

.965 

WK 

351 

54.20 

6.358 

390 

54.58 

6.573 

-.798 

.425 

PC 

351 

54.92 

5.829 

388 

55.34 

5.579 

-1.000 

.317 

MK 

349 

56.13 

5.733 

389 

56.56 

5.590 

-1.031 

.303 

El 

346 

56.64 

7.879 

388 

57.68 

7.605 

-1.818 

.069 

AS 

345 

55.23 

8.288 

387 

55.21 

7.806 

.034 

.973 

MC 

344 

58.16 

7.566 

387 

58.26 

7.713 

-.177 

.860 

AO 

308 

57.92 

6.768 

356 

58.42 

7.182 

-.919 

.359 

VE 

351 

54.71 

5.716 

388 

55.10 

5.836 

-.916 

.361 

AFQT 

348 

68.94 

16.985 

388 

70.00 

16.544 

-.857 

.392 

Table  A-8 

Record  data  for  minority 


Those  who  took  web- 
based  AFCT 

Those  who  took  paper- 
based  AFCT 

T 

P 

N 

Mean 

SD 

N 

Mean 

SD 

GS 

216 

51.00 

7.623 

166 

52.57 

7.720 

-1.984 

.048 

AR 

217 

53.52 

6.317 

163 

53.34 

6.757 

.267 

.790 

WK* 

218 

49.61 

6.875 

159 

52.21 

6.842 

-3.962 

.000 

PC* 

216 

52.06 

6.084 

158 

54.10 

6.011 

-3.219 

.001 

MK* 

218 

55.82 

4.997 

158 

56.28 

5.746 

6.864 

.000 

El 

217 

51.99 

8.319 

158 

53.27 

8.979 

-1.423 

.156 

AS 

216 

47.27 

7.944 

158 

48.74 

8.161 

-1.747 

.081 

MC 

212 

52.06 

7.930 

158 

53.50 

7.685 

-1.751 

.081 

AO 

188 

55.88 

6.986 

141 

56.66 

7.294 

-.983 

.326 

VE* 

217 

50.57 

6.231 

158 

53.12 

6.134 

-3.939 

.000 

AFQT 

215 

58.01 

17.647 

158 

63.32 

19.258 

-2.762 

.006 

The  testing  found  significant  differences  for  majority  personnel  in  two  subtests:  PC 
and  AO  (see  Table  A-9).  There  were  no  significant  differences  in  the  two  groups  prior  to 
our  testing,  so  a  possible  source  of  this  difference  in  our  tests  is  due  to  the  different 
administration  modes. 


Table  A-9 

Testing  data  for  majority 


Those  who  took  web- 
based  AFCT 

Those  who  took  paper- 
based  AFCT 

T 

P 

N 

Mean 

SD 

N 

Mean 

SD 

GS 

352 

55.23 

7.004 

395 

55.70 

7.465 

-.884 

.377 

AR 

350 

52.61 

7.788 

394 

52.97 

7.412 

-.646 

.519 

WK 

351 

55.07 

6.369 

390 

55.14 

7.304 

-.138 

.890 

PC* 

351 

51.74 

7.549 

388 

54.05 

6.833 

-4.366 

.000 

MK 

349 

53.83 

6.714 

389 

54.63 

6.980 

-1.583 

.114 

El 

346 

59.53 

11.616 

388 

59.66 

11.256 

-.154 

.878 

AS 

345 

54.12 

9.691 

387 

53.94 

9.411 

.255 

.799 

MC 

344 

57.51 

8.418 

387 

58.93 

8.721 

-2.233 

.026 

AO* 

308 

52.63 

10.134 

356 

56.08 

9.085 

-4.625 

.000 

VE 

351 

4.13 

6.394 

388 

55.09 

7.052 

-1.931 

.054 

AFQT 

348 

62.88 

20.632 

388 

66.28 

20.346 

-2.248- 

.025 

Four  tests  show  significant  differences  between  the  two  minority  groups,  two  of 
which  were  not  pre-existing.  Significant  differences  were  found  for  PC  and  VE,  as  was 
seen  in  the  record  data,  but  the  differences  in  WK  and  MK  no  longer  exist  (see  Table 
A-10).  In  addition,  scores  for  MC  and  AO  are  significantly  different,  possibly  indicating  a 
difference  between  modes. 
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Table  A- 10 

Testing  data  for  minority 


Those  who  took  web- 
based  AFCT 

Those  who  took  paper- 
based  AFCT 

T 

P 

N 

Mean 

SD 

N 

Mean 

SD 

GS 

216 

50.61 

7.420 

166 

52.06 

8.227 

-1.806 

.072 

AR 

217 

51.14 

7.447 

163 

50.18 

8.221 

1.189 

.235 

WK 

218 

50.52 

7.946 

159 

52.01 

8.183 

-1.943 

.053 

PC* 

216 

49.09 

7.976 

158 

52.39 

7.127 

-4.132 

.000 

MK 

218 

52.03 

6.441 

158 

52.50 

7.173 

-.623 

.534 

El 

217 

52.66 

11.507 

158 

55.33 

11.601 

-2.211 

.028 

AS 

216 

46.19 

8.776 

158 

46.68 

9.216 

-.522 

.602 

MC* 

212 

51.78 

8.580 

158 

54.66 

8.688 

-3.177 

.002 

AO* 

188 

50.15 

10.448 

141 

53.71 

9.854 

-3.133 

.002 

VE* 

217 

49.96 

7.689 

158 

52.39 

7.605 

-3.036 

.003 

AFQT 

215 

51.86 

22.497 

158 

56.46 

23.062 

-1.931 

.054 

Summary 


The  two  groups  that  participated  in  this  equating  study  were  slightly  different  prior 
to  their  participation.  As  the  tables  above  show,  male  and  majority  personnel  were 
equivalent,  but  overall  examinees  as  well  as  females  and  minority  primarily  tended  to 
differ  on  the  verbal  tests  (WK  and  PC,  as  well  as  the  composite  VE). 

The  results  of  the  equating  study  show  new  statistically  significant  differences, 
especially  in  MC  and  AO.  Because  these  differences  did  not  exist  between  the  groups 
prior  to  testing,  it  is  feasible  that  these  differences  are  due  to  the  mode  of  administration 
of  the  test. 


Appendix  B: 

Selected  Portions  of  the  Manual  for  Administration 
Armed  Forces  Classification  Test  (AFCT) 

Version  19G 
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5.  Testing  Materials. 

Testing  materials  required  for  each  examinee  are: 

a.  Computer 

b.  Privacy  Act  Statement 

c.  Scratch  paper  -  2  sheets 

d.  Two  number  2  pencils  (with  eraser) 

Materials  required  for  the  Test  Administrator  are: 

a.  This  manual 

b.  Sufficient  computers 

c.  Sufficient  copies  of  the  Privacy  Act  Statement 

d.  Sufficient  number  2  pencils  with  erasers 

e.  Sufficient  scratch  paper 
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7.  Preparation  of  computers  prior  to  testing. 

Computers  should  already  be  set  up  prior  to  testing,  but  below  are  key  steps  if 
they  aren’t: 

1.  Screen  saver  should  be  off 

2.  Sound  should  be  off 

3.  In  Internet  Explorer,  no  toolbars  (Standard  Buttons,  Address  Bar,  Links) 
should  be  on,  ONLY  THE  STATUS  BAR  should  be  showing  (bottom  of  the 
screen).  Be  sure  IE  window  is  the  full  screen. 

Computers  need  to  be  networked  with  the  server  hosting  the  test  prior  to  continuing: 

1.  Turn  the  server  on.  You  don’t  have  to  log  into  the  machine,  just  turn  it  on. 

2.  Turn  the  computers  on.  Just  hit  enter  to  login. 

3.  Open  Internet  Explorer.  It  should  default  to  the  test  home  page,  which  says 
AECT  and  asks  for  a  password. 

4.  Minimize  Internet  Explorer. 
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Section  II,  part  2  for  paper: 


This  is  a  research  project  on  the  AFCT,  the  version  of  the  ASVAB  administered  while  in 
the  Navy.  As  this  is  a  research  project,  you  may  choose  to  not  do  the  test  or  stop  doing 
the  test  during  your  time  here,  and  there  will  be  no  penalty  to  you.  However,  if  you  do 
the  test  and  you  score  better  than  your  previous  ASVAB  scores,  your  personnel  record 
will  be  updated  to  reflect  the  higher  scores.  If  your  scores  are  worse,  your  current  scores 
will  remain  in  your  personnel  file.  The  goal  of  this  research  is  to  improve  the  AFCT,  so 
please  do  your  best  on  the  test. _ 


If  you  choose  to  take  the  test,  you  are  required  to  disclose  your  Social  Security  Number 
(SSN).  Your  SSN  will  be  used  to  verify  that  the  scores  you  make  are  correctly  transcribed 
to  a  scoring  worksheet.  Any  data  you  provide  is  For  Official  Use  Only  and  will  be 
maintained  and  used  in  strict  confidence  in  accordance  with  Federal  Laws  and 
Regulations. _ 


Now  look  at  the  sheet  entitled  Privacy  Act  Statement.  Read  the  Privacy  Act  statement 
silently  as  I  read  it  out  loud: 

Privacy  Act  Statement 

AUTHORITY:  10  USC  133  and  3013;  E.O.  9397 

PRINCIPAL  PURPOSE:  To  collect  and  measure  an  individual’s  aptitude  for  re¬ 
enlistment,  re-classification,  or  training  as  a  commissioned  or  warrant  officer  for 
assignments  to  various  military  positions. 

ROUTINE  USES:  The  scores  an  individual  makes  (if  higher  than  their  previous  ASVAB 
scores)  will  be  transcribed  onto  the  appropriate  military  personnel  records  and 
furnished  to  evaluation  boards  and  officials. 

DISCLOSURE:  If  you  choose  to  participate,  completing  this  form  is  mandatory.  Your 
social  security  number  is  used  to  verify  that  the  score  you  make  is  correctly  transcribed 
on  your  military  personnel  record. 

I  certify  that  I  am  physically  and  mentally  fit  and  that  I  have  neither  given  nor  received 
unauthorized  assistance  in  conjunction  with  this  test. 


Now  read  the  final  statement. 


Eor  PAPER  ONLY: 


In  the  upper  center  portion  of  the  answer  sheet  there  is  a  black  printed  sequence 
number.  Write  that  number  on  the  Privacy  Act  Statement  in  the  blank  for 
Sequence/Computer  Number. _ 
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Print  your  last  name  and  Social  Security  Number  in  the  blanks  and  sign  in  the  space 
provided. _ 


Section  II,  part  2  for  WEB: 


Since  this  test  is  computer-based,  we'll  take  the  first  few  minutes  so  that  you  can  get 
used  to  the  mouse  on  the  laptop  in  front  of  you.  The  large  area  below  the  letter  keys  is 
the  mouse  pad  and  your  finger  is  the  mouse.  When  you  touch  that  area,  moving  your 
finger  causes  the  arrow  on  the  screen  to  move. 

In  this  test  you’ll  need  to  select  answers  by  left  clicking.  To  do  this,  use  the  left  button 
just  below  the  "mouse  pad".  Do  not  use  the  grey  circle  with  arrows. 

There  are  three  games  showing  on  the  desktop  -  solitaire,  spider  solitaire,  and 
minesweeper.  Spend  the  next  few  minutes  playing  the  games  so  you  can  get  better 
acquainted  with  how  to  use  the  mouse.  I'm  going  to  walk  around  and  make  sure 
everyone  understands  how  to  use  the  mouse  before  we  begin  the  test.  If  you  need  to  take 
a  head  break,  now  would  be  the  time  to  do  it.  The  test  may  take  2-3  hours. _ 


Walk  around  the  room,  checking  on  each  person  to  be  sure  they  can  use  the  mouse. 


I  am  (give  your  name),  your  Test  Administrator  today.  I  will  be  administering  the  Armed 
Forces  Classification  Test  (AFCT). 


This  is  a  research  project  on  the  AFCT,  the  version  of  the  ASVAB  administered  while  in 
the  Navy.  As  this  is  a  research  project,  you  may  choose  to  not  do  the  test  or  stop  doing 
the  test  during  your  time  here,  and  there  will  be  no  penalty  to  you.  However,  if  you  do 
the  test  and  you  score  better  than  your  previous  ASVAB  scores,  your  personnel  record 
will  be  updated  to  reflect  the  higher  scores.  If  your  scores  are  worse,  your  current  scores 
will  remain  in  your  personnel  file.  The  goal  of  this  research  is  to  improve  the  AFCT,  so 
please  do  your  best  on  the  test. _ 


If  you  choose  to  take  the  test,  you  are  required  to  disclose  your  Social  Security  Number 
(SSN).  Your  SSN  will  be  used  to  verify  that  the  scores  you  make  are  correctly  transcribed 
to  a  scoring  worksheet.  Any  data  you  provide  is  For  Official  Use  Only  and  will  be 
maintained  and  used  in  strict  confidence  in  accordance  with  Federal  Laws  and 
Regulations. _ 


Please  clear  your  desk  of  everything  but  the  laptop.  Each  of  you  will  receive  scratch 
paper  and  2  pencils.  Do  not  begin  the  test  until  you  are  told  to  do  so. _ 


Distribute  2  sheets  of  scratch  paper,  2  pencils,  and  the  Privacy  Act  Statement  to  each 
examinee,  then  say: 


On  these  tests  YOU  SHOULD  WORK  AS  RAPIDLY  AND  ACCURATELY  AS  YOU  CAN. 
DO  YOUR  OWN  WORK  AND  LISTEN  CAREEULLY  TO  INSTRUCTIONS  WHEN  THEY 
ARE  GIVEN.  Please  work  quietly  out  of  consideration  for  others  who  are  also  being 
tested. 


Now  look  at  the  sheet  entitled  Privacy  Act  Statement.  Read  the  Privacy  Act  statement 
silently  as  I  read  it  out  loud: 

Privacy  Act  Statement 

AUTHORITY:  lo  USC  133  and  3013;  E.O.  9397 

PRINCIPAL  PURPOSE:  To  collect  and  measure  an  individual’s  aptitude  for  re¬ 
enlistment,  re-classification,  or  training  as  a  commissioned  or  warrant  officer  for 
assignments  to  various  military  positions. 

ROUTINE  USES:  The  scores  an  individual  makes  (if  higher  than  their  previous  ASVAB 
scores)  will  be  transcribed  onto  the  appropriate  military  personnel  records  and 
furnished  to  evaluation  boards  and  officials. 

DISCLOSURE:  If  you  choose  to  participate,  completing  this  form  is  mandatory.  Your 
social  security  number  is  used  to  verify  that  the  score  you  make  is  correctly  transcribed 
on  your  military  personnel  record. 

I  certify  that  I  am  physically  and  mentally  fit  and  that  I  have  neither  given  nor  received 
unauthorized  assistance  in  conjunction  with  this  test. 


On  the  back  of  the  laptop  screen  is  a  number  written  on  a  yellow  square.  Write  that 
number  on  the  Privacy  Act  Statement  in  the  blank  for  Sequence/Computer  Number. 


Print  your  last  name  and  Social  Security  Number  in  the  blanks  and  sign  in  the  space 
provided. 


You  will  be  selecting  answers  on  the  laptop  in  front  of  you.  Use  the  scratch  paper  which 
was  given  to  you  for  any  figuring  you  need  to  do.  Return  this  scratch  paper  with  your 
other  materials  when  you  finish  the  test.  Calculators  of  ANY  kind  are  not  allowed  on  the 
test. 


The  AECT  contains  9  tests.  Each  test  has  its  own  instructions  and  time  limit,  which  you 
will  see  before  each  test.  When  you  finish  a  test  you  may  check  your  work  in  that  test 
ONLY.  Once  you  are  completely  done  with  a  test,  you  may  continue  on  to  the  next  test. 
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For  each  question,  be  sure  to  pick  the  BEST  ONE  of  the  possible  answers  listed.  When 
you  have  decided  which  one  of  the  choices  given  is  the  best  answer  to  a  question,  select 
that  answer  by  clicking  in  the  circle  in  front  of  that  choice. 


Answer  as  many  questions  as  possible.  Do  not  spend  too  much  time  on  any  one 
question.  Work  QUICKLY,  but  work  ACCURATELY. _ 


Each  test  is  timed,  and  the  time  remaining  will  be  shown  in  the  bottom  left  corner  of  the 
screen.  If  you  need  additional  scratch  paper  or  pencils  at  any  time  during  the  test,  please 
raise  your  hand  and  I  will  bring  them  to  you. 

Are  there  any  questions? 


Now,  maximize  Internet  Explorer  and  log  into  the  test  using  the  password  nprst2006. 
Each  test  begins  with  instructions  that  are  not  timed.  Read  through  each  set  of 
instructions  before  beginning  the  test.  Read  through  the  instructions  for  part  i  now,  and 
then  enter  your  SSN  in  the  box  at  the  bottom  before  clicking  the  Begin  button  to  start 
the  test. 


Examinees  will  raise  their  hands  when  they  are  done.  Press  the  power  button  (machine 
will  hibernate),  collect  all  materials,  and  then  quietly  dismiss  the  individual. 
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AFCT  Q&A 


Q;  What  if  I  suspect  someone  is  cheating? 

A:  Hover  around.  If  you  definitely  notice  someone  is  cheating,  at  the  end  of  the  section,  ask  them 
to  pack  up  their  materials,  take  back  the  Privacy  Act,  and  note  the  incident  on  the  form. 

Q;  What  if  someone  decides  that  they  don’t  want  to  continue  the  test? 

A:  Take  their  materials  and  thank  them  for  their  time.  Note  the  incident  on  their  Privacy  Act. 

Q;  What  if  someone  has  a  question  about  one  of  their  test  questions? 

A:  We’re  not  allowed  to  answer  any  questions  concerning  the  test. 

Q;  In  the  computer  administration  of  the  test,  what  if  someone  inadvertently  gets  out  of  the  test 
(closes  Internet  Explorer)  or  can’t  continue? 

A:  Go  back  into  Internet  Explorer  and  have  them  start  from  the  beginning.  After  entering  in  their 
SSN  and  clicking  Next  through  the  first  test  (stopping  after  pressing  the  SUBMIT  button),  go  to 
View  -  Toolbars  -  Address  Bar.  After  the  http://164.94. 7.  ,  type  in  the  filename  of  the  section 

they  should  be  going  to: 

Part  2  -  afct2.asp 

Part  3  -  afctS.asp 

Part  4  -  afct4.asp 

Etc 

Close  the  Address  Bar  again  (View  -  Toolbars  -  de-check  Address  Bar).  Make  note  of  their  SSN 
and  what  happened. 

Q:  Eor  computer  administration  of  the  test,  what  if  one  of  the  computers  just  dies? 

A:  Each  room  should  have  backups  (they  will  have  a  yellow  sticky  that  says  B_)  that  should  be 
“plug  and  play”.  Close  the  inoperative  computer,  disconnect  the  power  and  the  network  cable 
from  the  back  of  the  inoperative  computer,  and  plug  the  cables  into  the  backup  machine.  Turn 
the  machine  on,  open  Internet  Explorer,  and  continue  on  as  listed  in  the  EAQ  above.  Make  note 
of  what  happened,  including  what  computer  became  inoperative  and  what  backup  you  used,  as 
well  as  noting  who  it  was  and  what  section  they  went  to. 

Q:  How  do  we  find  out  our  scores? 

A:  Since  this  is  part  of  a  research  project,  the  tests  until  we  have  all  our  data  (likely  a  few 
months).  If  you  haven’t  heard  anything  before  December,  contact  Zannette.uriell@naw.mil . 

Q:  What  if  I  don’t  want  to  be  reclassified? 

A:  Respondents  are  not  automatically  reclassified  based  upon  the  test  scores. 


Appendix  C: 
Privacy  Act  Statement 


C-o 


Privacy  Act  Statement 


AUTHORITY:  10  USC  133  and  3013;  E.O.  9397 

PRINCIPAL  PURPOSE;  To  collect  and  measure  an  individual’s  aptitude  for  re¬ 
enlistment,  re-classification,  or  training  as  a  commissioned  or  warrant  officer  for 
assignments  to  various  military  positions. 

ROUTINE  USES:  The  scores  an  individual  makes  (if  higher  than  their  previous  ASVAB 
scores)  will  be  transcribed  onto  the  appropriate  military  personnel  records  and  furnished 
to  evaluation  boards  and  officials. 

DISCLOSURE:  If  you  choose  to  participate,  completing  this  form  is  mandatory.  Your 
social  security  number  is  used  to  verify  that  the  score  you  make  is  correctly  transcribed 
on  your  military  personnel  record. 

I  certify  that  I  am  physically  and  mentally  fit  and  that  I  have  neither  given  nor  received 
unauthorized  assistance  in  conjunction  with  this  test. 


Sequence/Computer  Number 


Last  Name 


Social  Security  No. 


SIGNATURE 


C-i 


Appendix  D: 
Scoring 


D-o 


AFCT  Scoring  in  SAS 


options  center  pageno=l  nodate  ls=124  ps=66;  /*for  PDF  ls=129,  ps=71*/ 
data  _null_;  call  symput ( ' cdate ' ,  put (date (),  weekdatelT . ) ) ;  run; 
data  _null_;  call  symput (' ctime ' ,  put (time (),  timeS.));  run; 

footnotel  'Source:  Navy  Personnel  Research,  Studies  &  Technology,  NPRST/PERS- 

1'  ; 

footnote2  "David  L.  Alderton,  Ph . D .  901/874-4633;  run  Sedate,  at  Setime.  "; 

^■k-k-k-k-k-k-k-k-k-k-k-k-k-k-k-k-k-k-k-k-k-k-k-k-k-k-k-k-k-k-k-k-k-k-k-k-k-k-k-k-k-k-k-k-k-k-k-k-k-k-k-k-k-k-k-k-k-k-k-k-k-k-k-k-k-k-k-k-k-k-k-k-k-k-k-k 
kkkkkkkkkkkkkkkkkkkk j 

/*This  will  perform  the  IRT  scoring  of  the  subtests...  but  does  not  include 
version  differences  */ 

/*This  also  computes  the  standards  scores  from  thetas  and  the  AFQT  from 
standard  scores.  */ 

/*!  used  several  numerical  examples  to  test,  including  those  given  by  Dan  and 
some  others.  */ 

/*There  are  no  controls  in  here  for  missing,  incomplete  or  messy  data  which 
needs  to  be  watched.*/ 

jkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkk 
kkkkkkkkkkkkkkkkkkkk j 

libname  afet  "Z:\ASVAB06\SCORING";  /*lib  name  for  my  computer...*/ 

jkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkj 

/*isolate  parms,  rename  them,  and  merge  them  together...*/ 

/*!  just  cut  and  pasted  this  from  the  Excel  file.  I  only*/ 

/*have  the  version  1  data  (top  of  file)  but  it  would  be  */ 

/*easy  to  modify  for  both.  */ 

jkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkj 

data  afet. parms; 


input 

.  subtest  $2.  inum  a  b  c; 

cards 

r 

GS 

1 

0.91436 

-0.78555 

0.14865 

GS 

2 

0.47381 

-2 . 64284 

0.18462 

GS 

3 

0.62907 

-2 . 93499 

0.20101 

GS 

4 

0.72723 

-1.24738 

0.2784 

GS 

5 

0.58742 

-1.47272 

0.27789 

GS 

6 

0.88309 

-0.99238 

0.20458 

GS 

7 

0.39527 

-2 . 65297 

0.25698 

GS 

8 

0.62042 

-0.94736 

0.17202 

GS 

9 

0.62517 

0.01806 

0.37433 

GS 

10 

0.79165 

-0.3915 

0.25134 

GS 

11 

1.018  -0 

.29386  0. 

32875 

GS 

12 

0.73272 

-0.90835 

0.1738 

GS 

13 

0.65149 

-0.01085 

0.06584 

GS 

14 

0.94343 

0.1288 

0.21421 

GS 

15 

0.71347 

0.69584 

0.14951 

GS 

16 

0.67643 

0.24637 

0.18816 

GS 

17 

0.49969 

0.06832 

0.21698 

GS 

18 

0.66381 

0.42804 

0.14121 

GS 

19 

0.75412 

1.27854 

0.38795 

GS 

20 

0.51658 

0.72605 

0.31128 

GS 

21 

0.40946 

0.60364 

0.14844 

GS 

22 

0.95975 

1.56912 

0.0843 

GS 

23 

1.085  1. 

86188  0. 

23835 

D-1 


GS 

24 

0.62037 

2.28963 

0 

GS 

25 

0.8328 

1.44953 

0 

AR 

1 

0.69161 

-2.37083 

0 

AR 

2 

0.53495 

-2 . 80441 

0 

AR 

3 

1 . 06292 

-1 . 60186 

0 

AR 

4 

0.69981 

-0.8137 

0 

AR 

5 

0.83974 

-1 . 00857 

0 

AR 

6 

1.4909 

-0.17713 

0 

AR 

7 

0.9524 

-0.72321 

0 

AR 

8 

0.93455 

-0.89134 

0 

AR 

9 

0.97787 

-0.44061 

0 

AR 

10 

1.26494 

-0.27203 

0 

AR 

11 

0.97071 

-0.89672 

0 

AR 

12 

0.81582 

-0.53374 

0 

AR 

13 

1 . 05432 

-0.09479 

0 

AR 

14 

1.187  -0. 

.14862  0. 

16084 

AR 

15 

1.09468 

1.412  0. 

42045 

AR 

16 

0.51542 

-0.06039 

0 

AR 

17 

1 . 07013 

-0.05964 

0 

AR 

18 

1 .18848 

0.78574 

0 

AR 

19 

0.92195 

0.81313 

0 

AR 

20 

1 .10426 

0.79569 

0 

AR 

21 

1.25263 

0.66573 

0 

AR 

22 

1.22938 

1.06802 

0 

AR 

23 

0.87833 

-0.19689 

0 

AR 

24 

1 . 91382 

1.07465 

0 

AR 

25 

1 .18448 

1 . 12197 

0 

AR 

26 

1 . 12751 

1 . 13302 

0 

AR 

27 

2 . 02064 

1.2274 

0 

AR 

28 

1 . 10695 

1 . 12523 

0 

AR 

29 

1.23125 

1.50646 

0 

AR 

30 

2 . 05102 

1.33308 

0 

WK 

1 

0.94623 

-1 . 99913 

0 

WK 

2 

1 . 02139 

-2.42811 

0 

WK 

3 

0.95334 

-2.70973 

0 

WK 

4 

0.93721 

-2.08811 

0 

WK 

5 

1 .10676 

-1 .81998 

0 

WK 

6 

1.09404 

-1.4362 

0 

WK 

7 

0.77163 

-2.42383 

0 

WK 

8 

1.53072 

-1.22798 

0 

WK 

9 

0.88658 

-1 . 60389 

0 

WK 

10 

1 .19039 

-1 . 60244 

0 

WK 

11 

1.24316 

-2.0244 

0 

WK 

12 

1.29483 

-1.53161 

0 

WK 

13 

1 . 02709 

-0.66848 

0 

WK 

14 

1.00874 

-0.49049 

0 

WK 

15 

1.39306 

-1.26894 

0 

WK 

16 

1.2093 

-0.99866 

0 

WK 

17 

0.85414 

-1.50791 

0 

WK 

18 

0.71445 

-1.07432 

0 

WK 

19 

1.3733 

-0.31955 

0 

WK 

20 

0.85212 

-0.60071 

0 

WK 

21 

1 . 12563 

-0.08497 

0 

WK 

22 

1.58637 

-0.23732 

0 

WK 

23 

1.38671 

-0.03478 

0 

WK 

24 

1 . 62807 

0.40636 

0 

WK 

25 

0.55365 

-0.18213 

0 

08836 

11436 

17583 

15007 

07038 

10938 

08162 

15836 

14032 

16704 

02216 

19634 

05369 

1798 

13462 


12794 

24954 

19108 

37857 

21308 

26127 

20573 

18008 

28383 

1856 

32535 

2139 

26605 

25046 

14229 

06185 

16112 

14682 

07391 

12403 

14317 

11548 

23881 

06828 

19522 

13798 

27972 

13248 

154 

22801 

25079 

09912 

21296 

23958 

35207 

5 

29288 

41813 

17273 

13315 


-2 


WK 

26 

WK 

27 

WK 

28 

WK 

29 

WK 

30 

WK 

31 

WK 

32 

WK 

33 

WK 

34 

WK 

35 

PC 

1 

PC 

2 

PC 

3 

PC 

4 

PC 

5 

PC 

6 

PC 

7 

PC 

8 

PC 

9 

PC 

10 

PC 

11 

PC 

12 

PC 

13 

PC 

14 

PC 

15 

MK 

1 

MK 

2 

MK 

3 

MK 

4 

MK 

5 

MK 

6 

MK 

7 

MK 

8 

MK 

9 

MK 

10 

MK 

11 

MK 

12 

MK 

13 

MK 

14 

MK 

15 

MK 

16 

MK 

17 

MK 

18 

MK 

19 

MK 

20 

MK 

21 

MK 

22 

MK 

23 

MK 

24 

MK 

25 

El 

1 

El 

2 

El 

3 

El 

4 

El 

5 

El 

6 

El 

7 

1.59244 
1 . 68835 
1 . 03606 
0.91295 
1 .16759 
1.58561 
1 .10822 
0.84336 
0.7019 
0.70198 
0.93677 
0.68608 
0.9056 
1.31657 
0.65822 
0.47424 
0.71842 
1.0989 
0.52058 
1 . 03986 
0.57626 
0.75584 
1 .16951 
1.53557 
1.52707 
0.48608 
0.75602 
0.7061 
0.83788 
1.03099 
0.91807 
1 .11046 
1.37977 
1 . 15755 
1.00277 
1.07472 
0.76672 
1.39979 
1 . 14591 
0.71741 
0.83677 
1.39926 
0.8545 
1.07006 
1.03234 
1.28953 
1 . 17254 
0.90738 
1 .8991 
1.55167 
0.54128 
0.44512 
0.7749 
1 . 08727 
0.57896 
0.63905 
0.61368 


0.44838 

0.14143 

-0.35464 

1.07383 

0.85341 

0.88667 

0.7892 

1.74273 

0.80273 

0.7401 

-1.30059 

-1 . 13724 

-1.33379 

-0.6578 

-1.42087 

-1.77563 

-1 .18292 

-0.57203 

-0.76966 

-1 .86214 

-0.93273 

0.31241 

0.64189 

0.15254 

0.27197 

-2.27829 

-1 . 65131 

-2.51714 

-0.20459 

-0.54983 

-0.85209 

0.09746 

-0.14145 

0.09751 

-0.12153 

-0.31053 

0.39425 

0.47199 

0.49367 

0.08486 

1.07672 

0.72736 

0.33316 

0.6619 

0.66915 

0.25302 

1 .1191 

0.85672 

1 . 15848 

1.29933 

-1.40113 

-2 .19275 

-0.30943 

0.49554 

-0.3041 

0.12713 

-0.13805 


0.31194 

0.36996 

0.25983 

0.22502 

0.31975 

0.36568 

0.11174 

0.16347 

0.36277 

0.18915 

0.18034 

0.14735 

0.1188 

0.11826 

0.07882 

0.07943 

0.10859 

0.11593 

0.14157 

0.12344 

0.13629 

0.1931 

0.15481 

0.17975 

0.20495 

0.22232 

0.15721 

0.15603 

0.21893 

0.18116 

0.12924 

0.40807 

0.19052 

0.04072 

0.26868 

0.12026 

0.04302 

0.27006 

0.30219 

0.07646 

0.20505 

0.48806 

0.14661 

0.08507 

0.17034 

0.08505 

0.14822 

0.08214 

0.21171 

0.22083 

0.09168 

0.16594 

0.19301 

0.39446 

0.06633 

0.31333 

0.20155 


El 

8 

El 

9 

El 

10 

El 

11 

El 

12 

El 

13 

El 

14 

El 

15 

El 

16 

El 

17 

El 

18 

El 

19 

El 

20 

AS 

1 

AS 

2 

AS 

3 

AS 

4 

AS 

5 

AS 

6 

AS 

7 

AS 

8 

AS 

9 

AS 

10 

AS 

11 

AS 

12 

AS 

13 

AS 

14 

AS 

15 

AS 

16 

AS 

17 

AS 

18 

AS 

19 

AS 

20 

AS 

21 

AS 

22 

AS 

23 

AS 

24 

AS 

25 

MC 

1 

MC 

2 

MC 

3 

MC 

4 

MC 

5 

MC 

6 

MC 

7 

MC 

8 

MC 

9 

MC 

10 

MC 

11 

MC 

12 

MC 

13 

MC 

14 

MC 

15 

MC 

16 

MC 

17 

MC 

18 

MC 

19 

0.43312 

0.63808 

0.59113 

0.40447 

1 . 13622 

0.74173 

1.36233 

0.61123 

1.22852 

3.39299 

1.35882 

1.40809 

1 . 65236 

0.72604 

0.94108 

1.02158 

0.88189 

0.41232 

0.96361 

2 . 04821 

0.52229 

1.32283 

0.78761 

0.79827 

2.01073 

0.64198 

0.67502 

0.35589 

0.83379 

0.96923 

2 . 05555 

0.44447 

1.31577 

1 . 12678 

0.95009 

1.07364 

1 .86967 

1 . 69923 

0.74279 

0.80909 

0.97807 

0.58193 

0.5678 

1.06938 

0.55806 

0.84652 

0.86135 

0.81386 

0.72131 

1 . 15143 

0.60165 

0.86743 

0.99059 

0.55576 

0.77001 

1 . 08954 

0.71698 


-2 .16243 
-0.6724 
0.23785 
-1.44321 
0.93676 
1 . 10333 
0.97357 
0.75783 
1 .10599 
2.54553 
1.49817 
1 . 92177 
2 . 0484 
-0.81937 
-1.23619 
-1 . 08821 
-0.31706 
-0.52765 
0.0404 
0.56323 
-0.11796 
0.76135 
0.96572 
0.01639 
1 .09202 
0.39584 
1 . 10998 
-0.11162 
0.91499 
0.57023 
0.68303 
0.43339 
1.02899 
1.3408 
1.36492 
1 . 67572 
1 . 08781 
1.27657 
0.18245 
-0.48081 
-1.08137 
-0.5386 
0.45434 
0.08375 
-0.91499 
0.20709 
0.6525 
0.89453 
0.40877 
0.78216 
-0.0506 
0.29763 
1.75212 
0.23291 
0.53024 
1.74088 
0.33447 


0.0974 

0.10262 

0.16543 

0.10542 

0.18266 

0.10825 

0.26708 

0.11334 

0.21568 

0.23674 

0.17364 

0.28898 

0.14688 

0.18747 

0.22178 

0.1033 

0.16216 

0.08818 

0.11989 

0.4015 

0.14273 

0.32233 

0.48112 

0.11234 

0.49381 

0.03703 

0.5 

0.07506 

0.07623 

0.17139 

0.26783 

0.07009 

0.18656 

0.21803 

0.16984 

0.15328 

0.22078 

0.1821 

0.16946 

0.24885 

0.09415 

0.07614 

0.11108 

0.11457 

0.29483 

0.08688 

0.33261 

0.26505 

0.12841 

0.39641 

0.17606 

0.04723 

0.18112 

0.11014 

0.09702 

0.34716 

0.22632 


MC 

20 

0.79906 

1 . 12321 

0.20126 

MC 

21 

0.56447 

0.69682 

0.03684 

MC 

22 

1.39813 

0.84086 

0.25107 

MC 

23 

0.8079 

1.34906 

0.1776 

MC 

24 

0.87742 

1 . 83394 

0.18349 

MC 

25 

0.79277 

1.02826 

0.22463 

AO 

1 

0.92604 

0.27877 

0.41604 

AO 

2 

1 . 63548 

-0.40704 

0.37784 

AO 

3 

1.3188 

-0.44956 

0.39579 

AO 

4 

0.98753 

-0.42992 

0.30135 

AO 

5 

1.25043 

-0.19651 

0.33863 

AO 

6 

0.92736 

-0.26473 

0.4227 

AO 

7 

0.9625 

-0.26908 

0.42695 

AO 

8 

1.43875 

0.02796 

0.29233 

AO 

9 

1.25061 

0.14978 

0.27594 

AO 

10 

1.0184 

0.32411 

0.25874 

AO 

11 

0.55062 

1.26054 

0.27527 

AO 

12 

0.72643 

-0.15407 

0.29664 

AO 

13 

1.57116 

-1.49674 

0.18803 

AO 

14 

0.93175 

-1 .19936 

0.08545 

AO 

15 

1.34623 

-1.28518 

0.02998 

AO 

16 

0.88941 

-1 . 13276 

0.04437 

AO 

17 

1.46408 

-1 . 12058 

0.04654 

AO 

18 

0.90133 

-1 . 15894 

0.05187 

AO 

19 

0.68892 

-0.87208 

0.04935 

AO 

20 

1.42837 

-0.81359 

0.21848 

AO 

21 

1.20289 

-0.72515 

0.22876 

AO 

22 

1.53127 

-0.51003 

0.20649 

AO 

23 

0.89915 

-0.40272 

0.19925 

AO 

24 

1 .16498 

-0.30876 

0.21703 

AO 

25 

0.48491 

0.08342 

0.1219 

run; 

^■k-k-k-k-k-k-k-k-k-k-k-k-k-k-k-k-k-k-k-k-k-k-k-k-k-k-k-k-k-k-k-k-k-k-k-k-k-k-k-k-k-k-k-k-k-k-k^ 

/*begin  creating  the  1-record  parameter  file...*/ 

/*This  could  be  looped  by  didn't  seem  worth  it  */ 

^■kkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkk^ 

data  parms;  set  afct.parms;  run;  /*Load  afct.parms  into  memory, 
faster* / 

/*GS*/  /*create  and  label  the  a,  b,  c  parameters  for  easy  array 
proc  transpose  data=parms  out=GSa  prefix=GSa;  where  subtest='GS 

proc  transpose  data=parms  out=GSb  prefix=GSb;  where  subtest='GS 

proc  transpose  data=parms  out=GSc  prefix=GSc;  where  subtest='GS 

data  GSparms;  merge  GSa (drop=_name_)  GSb (drop=_name_)  GSc (drop= 
/*AR*/ 

proc  transpose  data=parms  out=ARa  prefix=ARa;  where  subtest='AR 

proc  transpose  data=parms  out=ARb  prefix=ARb;  where  subtest='AR 

proc  transpose  data=parms  out=ARc  prefix=ARc;  where  subtest='AR 

data  ARparms;  merge  ARa(drop=  name  )  ARb (drop=_name_)  ARc (drop= 
/*WK*/  ~  ~ 

proc  transpose  data=parms  out=WKa  prefix=WKa;  where  subtest='WK 

proc  transpose  data=parms  out=WKb  prefix=WKb;  where  subtest='WK 

proc  transpose  data=parms  out=WKc  prefix=WKc;  where  subtest='WK 

data  WKparms;  merge  WKa(drop=  name_)  WKb (drop=_name_)  WKc (drop= 
/*PC*/ 

proc  transpose  data=parms  out=PCa  prefix=PCa;  where  subtest='PC 

proc  transpose  data=parms  out=PCb  prefix=PCb;  where  subtest='PC 


.  a  little 

usage  later*/ 
;  var  a;  run; 
;  var  b;  run; 
;  var  c;  run; 
name_) ;  run; 

;  var  a;  run; 
;  var  b;  run; 
;  var  c;  run; 
name_) ;  run; 

;  var  a;  run; 
;  var  b;  run; 
;  var  c;  run; 
name_) ;  run; 

;  var  a;  run; 
;  var  b;  run; 


proc  transpose  data=parms  out=PCc  prefix=PCc;  where  subtest= ' PC ' ;  var  c;  run; 
data  PCparms;  merge  PCa(drop=  name  )  PCb (drop=_name_)  PCc (drop=  name_) ;  run; 
/*MK*/ 

proc  transpose  data=parms  out=MKa  prefix=MKa;  where  subtest= ' MK ' ;  var  a;  run; 

proc  transpose  data=parms  out=MKb  prefix=MKb;  where  subtest= ' MK ' ;  var  b;  run; 

proc  transpose  data=parms  out=MKc  prefix=MKc;  where  subtest= ' MK ' ;  var  c;  run; 

data  MKparms;  merge  MKa(drop=  name  )  MKb(drop=  name  )  MKc (drop=  name_) ;  run; 
/*EI*/ 

proc  transpose  data=parms  out=EIa  prefix=EIa;  where  subtest= ' El ' ;  var  a;  run; 

proc  transpose  data=parms  out=EIb  prefix=EIb;  where  subtest= ' El ' ;  var  b;  run; 

proc  transpose  data=parms  out=EIc  prefix=EIc;  where  subtest= ' El ' ;  var  c;  run; 

data  Elparms;  merge  EIa(drop=  name  )  Elb (drop=_name_)  EIc (drop=  name_) ;  run; 
/*AS*/ 

proc  transpose  data=parms  out=ASa  prefix=ASa;  where  subtest= ' AS ' ;  var  a;  run; 

proc  transpose  data=parms  out=ASb  prefix=ASb;  where  subtest= ' AS ' ;  var  b;  run; 

proc  transpose  data=parms  out=ASc  prefix=ASc;  where  subtest= ' AS ' ;  var  c;  run; 

data  ASparms;  merge  ASa(drop=  name  )  ASb(drop=  name_)  ASc (drop=_name  );  run; 
/*MC*/ 

proc  transpose  data=parms  out=MCa  prefix=MCa;  where  subtest= ' MC ' ;  var  a;  run; 

proc  transpose  data=parms  out=MCb  prefix=MCb;  where  subtest= ' MC ' ;  var  b;  run; 

proc  transpose  data=parms  out=MCc  prefix=MCc;  where  subtest= ' MC ' ;  var  c;  run; 

data  MCparms;  merge  MCa(drop=  name  )  MCb(drop=  name  )  MCc (drop=  name  );  run; 
/*A0...  not  really  necessary*/ 

proc  transpose  data=parms  out=AOa  prefix=AOa;  where  subtest= ' AO ' ;  var  a;  run; 

proc  transpose  data=parms  out=AOb  prefix=AOb;  where  subtest= ' AO ' ;  var  b;  run; 

proc  transpose  data=parms  out=AOc  prefix=AOc;  where  subtest= ' AO ' ;  var  c;  run; 

data  AOparms;  merge  AOa (drop=_name  )  AOb(drop=  name  )  AOc (drop=  name  );  run; 

/*Gather  into  1  record  of  684  variables  to  merge  with  individual  subject 
records*/ 

data  af ct . IRTparms ; 

merge  GSparms  ARparms  WKparms  PCparms  MKparms  Elparms  ASparms  MCparms 
AOparms ; 

GSitems=25;  ARitems=30;  WKitems=35;  PCitems=15;  MKitems=25; 

EIitems=20;  ASitems=25;  MCitems=25;  AOitems=25; 
run; 

proc  import  dataf ile="Z : \ASVAB06\data\afct  complete  SAS.csv"  out=WORK . ZSAS 

dbms=csv  replace; 

delimiter=", "; 

getnames=yes ; 

run; 

proc  sort  data=WORK . ZSAS ;  by  SSN;  run;  /*for  later  merges*/ 

proc  sql;  /*This  creates  a  Cartesian  join...  the  parmeters  record  joined  to 
each  person*/ 
create  table  WithParm  as 
select  * 

from  WORK. ZSAS,  afct . IRTparms ; 
quit; 

^■k-k-k-k-k-k-k-k-k-k-k-k-k-k-k-k-k-k-k-k-k-k-k-k-k-k-k-k-k-k-k-k-k-k-k-k-k-k-k-k-k-k-k-k-k-k-k-k-k-k-k-k-k-k-k-k-k-k-k-k-k-k-k-k-k-k-k-k-k^ 

/*The  next  steps  do  the  subtest  scoring  and  outputs  theta  and  ID  only*/ 

/*  this  could  have  been  done  in  a  macro  loop  but  it  was  quicker  to  */ 

/*  just  to  cut  and  paste,  replace  the  subtest  prefix  and  #  of  items.*/ 

/*  I  tested  this  code  against  numerical  examples  in  Dan's  PDF.  */ 

^■k'kk'kk'kkkk'kkkkkkkkkkkk'kkkkkkkk'kkkkkkkkkkkkkkkkkkkkkkkk'kkkkkkkkkkkkkkkk^ 

data  ScoreGS; 

set  WithParm  (keep=SSN  GSal-GSa25  GSbl-GSb25  GScl-GSc25  CGS01-CGS25 
GSitems) ; 
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nitems=25; 

/*initialize  constants*/ 
ta=-4;  tb=4;  fpd=0;  mu=0;  s=l; 

/*set  arrays*/ 
array  a (25)  GSal-GSa25; 
array  b(25)  GSbl-GSb25; 
array  c(25)  GScl-GSc25; 
array  u(25)  CGS01-CGS25; 

/*now  get  initial  estimate  of  fa  on  the  first  pass  through  the  data*/ 
do  i  =  1  to  nitems; 

expval=exp (a (i) *1 . 7* (ta-b (i) ) ) ;  /*note  in  first  call,  theta=ta*/ 
fpd  =  fpd  +  a(i)*1.7*(-l.  +  l./(l.  +  expval)  +  u(i)  - 
(c  (i) *u (i) ) / (c (i)  +  expval)); 
end; 

fa  =  fpd  +  (mu  -  ta) /s**2  ;  /*note  in  first  call,  theta=ta*/ 

/*first  part  worked...  and  matches  Dan's  cycle  0  value  for  fa=4. 30928*/ 
/*now  get  initial  estimate  of  fp  on  the  second  pass  through  the  data*/ 
/*second  part  worked...  and  matches  Dan's  cycle  1  value  for  fp=  -0.27936*/ 
/*now  wrap  it  in  the  15  cycle  iteration  for  final  theta*/ 
do  krep  =  1  to  15; 
tp= (ta+tb) /2 ; 
fpd=0  ; 

do  i  =  1  to  nitems; 

expval=exp (a (i) *1 . 7* (tp-b (i) ) ) ;  /*note  in  second  call,  theta=tp*/ 
fpd  =  fpd  +  a(i)*1.7*(-l.  +  l./(l.  +  expval)  +  u(i)  - 
(c (i) *u (i) ) / (c (i)  +  expval)); 
end; 

fp  =  fpd  +  (mu  -  tp) /s**2  ;  /*note  in  second  call,  theta=tp*/ 
if  fp  =  0  then  stop  ; 
else  if  fa*fp  >  0  then 
do; 

ta=tp;  fa=fp; 
end; 

else  tb=tp; 
end; 

GS_theta= (ta+tb) /2 ; 
keep  ssn  GS_theta  ; 
run; 

data  ScoreAR; 

set  WithParm  (keep=SSN  ARal-ARa30  ARbl-ARb30  ARcl-ARc30  CAR01-CAR30 
ARi terns ) ; 
nitems=30 ; 

/*initialize  constants*/ 
ta=-4;  tb=4;  fpd=0;  mu=0;  s=l; 

/*set  arrays*/ 
array  a  (30)  ARal-ARa30; 
array  b(30)  ARbl-ARb30; 
array  c(30)  ARcl-ARc30; 
array  u(30)  CAR01-CAR30; 

/*now  get  initial  estimate  of  fa  on  the  first  pass  through  the  data*/ 
do  i  =  1  to  nitems; 

expval=exp (a (i) *1 . 7* (ta-b (i) ) ) ;  /*note  in  first  call,  theta=ta*/ 
fpd  =  fpd  +  a(i)*1.7*(-l.  +  l./(l.  +  expval)  +  u(i)  - 
(c  (i) *u (i) ) / (c (i)  +  expval)); 
end; 

fa  =  fpd  +  (mu  -  ta) /s**2  ;  /*note  in  first  call,  theta=ta*/ 


/*first  part  worked...  and  matches  Dan's  cycle  0  value  for  fa=4. 30928*/ 
/*now  get  initial  estimate  of  fp  on  the  second  pass  through  the  data*/ 
/*second  part  worked...  and  matches  Dan's  cycle  1  value  for  fp=  -0.27936*/ 
/*now  wrap  it  in  the  15  cycle  iteration  for  final  theta*/ 
do  krep  =  1  to  15; 
tp= (ta+tb) /2 ; 
fpd=0  ; 

do  i  =  1  to  nitems; 

expval=exp (a (i) *1 . 7* (tp-b (i) ) ) ;  /*note  in  second  call,  theta=tp*/ 
fpd  =  fpd  +  a(i)*1.7*(-l.  +  l./(l.  +  expval)  +  u(i)  - 
(c (i) *u (i) ) / (c (i)  +  expval)); 
end; 

fp  =  fpd  +  (mu  -  tp) /s**2  ;  /*note  in  second  call,  theta=tp*/ 
if  fp  =  0  then  stop  ; 
else  if  fa*fp  >  0  then 
do; 

ta=tp;  fa=fp; 
end; 

else  tb=tp; 
end; 

AR_theta= (ta+tb) /2; 
keep  ssn  AR  theta  ; 
run; 

data  ScoreWK; 

set  WithParm  (keep=SSN  WKal-WKa35  WKbl-WKb35  WKcl-WKc35  CWK01-CWK35 
WKitems ) ; 
nitems=35 ; 

/*initialize  constants*/ 
ta=-4;  tb=4;  fpd=0;  mu=0;  s=l; 

/*set  arrays*/ 
array  a (35)  WKal-WKa35; 
array  b(35)  WKbl-WKb35; 
array  c(35)  WKcl-WKc35; 
array  u(35)  CWK01-CWK35; 

/*now  get  initial  estimate  of  fa  on  the  first  pass  through  the  data*/ 
do  i  =  1  to  nitems; 

expval=exp (a (i) *1 . 7* (ta-b (i) ) ) ;  /*note  in  first  call,  theta=ta*/ 
fpd  =  fpd  +  a(i)*1.7*(-l.  +  l./(l.  +  expval)  +  u(i)  - 
(c (i) *u (i) ) / (c (i)  +  expval)); 
end; 

fa  =  fpd  +  (mu  -  ta) /s**2  ;  /*note  in  first  call,  theta=ta*/ 

/*first  part  worked...  and  matches  Dan's  cycle  0  value  for  fa=4. 30928*/ 
/*now  get  initial  estimate  of  fp  on  the  second  pass  through  the  data*/ 
/*second  part  worked...  and  matches  Dan's  cycle  1  value  for  fp=  -0.27936*/ 
/*now  wrap  it  in  the  15  cycle  iteration  for  final  theta*/ 
do  krep  =  1  to  15; 
tp= (ta+tb) /2 ; 
fpd=0  ; 

do  i  =  1  to  nitems; 

expval=exp (a (i) *1 . 7* (tp-b (i) ) ) ;  /*note  in  second  call,  theta=tp*/ 
fpd  =  fpd  +  a(i)*1.7*(-l.  +  l./(l.  +  expval)  +  u(i)  - 
(c (i) *u (i) ) / (c (i)  +  expval)); 
end; 

fp  =  fpd  +  (mu  -  tp) /s**2  ;  /*note  in  second  call,  theta=tp*/ 
if  fp  =  0  then  stop  ; 
else  if  fa*fp  >  0  then 
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do; 


ta=tp;  fa=fp; 
end; 

else  tb=tp; 
end; 

WK_theta= (ta+tb) /2 ; 
keep  ssn  WK  theta  ; 
run; 

data  ScorePC; 

set  WithParm  (keep=SSN  PCal-PCal5  PCbl-PCbl5  PCcl-PCcl5  CPC01-CPC15 
PC items) ; 
nitems=15; 

/*initialize  constants*/ 
ta=-4;  tb=4;  fpd=0;  mu=0;  s=l; 

/*set  arrays*/ 
array  a (15)  PCal-PCal5; 
array  b(15)  PCbl-PCbl5; 
array  c(15)  PCcl-PCcl5; 
array  u(15)  CPC01-CPC15; 

/*now  get  initial  estimate  of  fa  on  the  first  pass  through  the  data*/ 
do  i  =  1  to  nitems; 

expval=exp (a (i) *1 . 7* (ta-b (i) ) ) ;  /*note  in  first  call,  theta=ta*/ 
fpd  =  fpd  +  a(i)*1.7*(-l.  +  l./(l.  +  expval)  +  u(i)  - 
(c (i) *u (i) ) / (c (i)  +  expval)); 
end; 

fa  =  fpd  +  (mu  -  ta) /s**2  ;  /*note  in  first  call,  theta=ta*/ 

/*first  part  worked...  and  matches  Dan's  cycle  0  value  for  fa=4. 30928*/ 
/*now  get  initial  estimate  of  fp  on  the  second  pass  through  the  data*/ 
/*second  part  worked...  and  matches  Dan's  cycle  1  value  for  fp=  -0.27936*/ 
/*now  wrap  it  in  the  15  cycle  iteration  for  final  theta*/ 
do  krep  =  1  to  15; 
tp= (ta+tb) /2 ; 
fpd=0  ; 

do  i  =  1  to  nitems; 

expval=exp (a (i) *1 . 7* (tp-b (i) ) ) ;  /*note  in  second  call,  theta=tp*/ 
fpd  =  fpd  +  a(i)*1.7*(-l.  +  l./(l.  +  expval)  +  u(i)  - 
(c (i) *u (i) ) / (c (i)  +  expval)); 
end; 

fp  =  fpd  +  (mu  -  tp) /s**2  ;  /*note  in  second  call,  theta=tp*/ 
if  fp  =  0  then  stop  ; 
else  if  fa*fp  >  0  then 
do; 

ta=tp;  fa=fp; 
end; 

else  tb=tp; 
end; 

PC_theta= (ta+tb) /2 ; 
keep  ssn  PC_theta  ; 
run; 

data  ScoreMK; 

set  WithParm  (keep=SSN  MKal-MKa25  MKbl-MKb25  MKcl-MKc25  CMK01-CMK25 
MKitems ) ; 
nitems=25; 

/*initialize  constants*/ 
ta=-4;  tb=4;  fpd=0;  mu=0;  s=l; 
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/*set  arrays*/ 
array  a (25)  MKal-MKa25; 
array  b(25)  MKbl-MKb25; 
array  c(25)  MKcl-MKc25; 
array  u ( 2  5 )  CMKO 1 -CMK2  5 ; 

/*now  get  initial  estimate  of  fa  on  the  first  pass  through  the  data*/ 
do  i  =  1  to  nitems; 

expval=exp (a (i) *1 . 7* (ta-b (i) ) ) ;  /*note  in  first  call,  theta=ta*/ 
fpd  =  fpd  +  a(i)*1.7*(-l.  +  l./(l.  +  expval)  +  u(i)  - 
(c (i) *u (i) ) / (c (i)  +  expval)); 
end; 

fa  =  fpd  +  (mu  -  ta) /s**2  ;  /*note  in  first  call,  theta=ta*/ 

/*first  part  worked...  and  matches  Dan's  cycle  0  value  for  fa=4. 30928*/ 
/*now  get  initial  estimate  of  fp  on  the  second  pass  through  the  data*/ 
/*second  part  worked...  and  matches  Dan's  cycle  1  value  for  fp=  -0.27936*/ 
/*now  wrap  it  in  the  15  cycle  iteration  for  final  theta*/ 
do  krep  =  1  to  15; 
tp= (ta+tb) /2 ; 
fpd=0  ; 

do  i  =  1  to  nitems; 

expval=exp (a (i) *1 . 7* (tp-b (i) ) ) ;  /*note  in  second  call,  theta=tp*/ 
fpd  =  fpd  +  a(i)*1.7*(-l.  +  l./(l.  +  expval)  +  u(i)  - 
(c (i) *u (i) ) / (c (i)  +  expval)); 
end; 

fp  =  fpd  +  (mu  -  tp) /s**2  ;  /*note  in  second  call,  theta=tp*/ 
if  fp  =  0  then  stop  ; 
else  if  fa*fp  >  0  then 
do; 

ta=tp;  fa=fp; 
end; 

else  tb=tp; 
end; 

MK_theta= (ta+tb) /2; 
keep  ssn  MK  theta  ; 
run; 

data  ScoreEI; 

set  WithParm  (keep=SSN  EIal-EIa20  EIbl-EIb20  EIcl-EIc20  CEI01-CEI20 
El items) ; 
nitems=20; 

/*initialize  constants*/ 
ta=-4;  tb=4;  fpd=0;  mu=0;  s=l; 

/*set  arrays*/ 
array  a(20)  EIal-EIa20; 
array  b(20)  EIbl-EIb20; 
array  c(20)  EIcl-EIc20; 
array  u(20)  CEI01-CEI20; 

/*now  get  initial  estimate  of  fa  on  the  first  pass  through  the  data*/ 
do  i  =  1  to  nitems; 

expval=exp (a (i) *1 . 7* (ta-b (i) ) ) ;  /*note  in  first  call,  theta=ta*/ 
fpd  =  fpd  +  a(i)*1.7*(-l.  +  l./(l.  +  expval)  +  u(i)  - 
(c (i) *u (i) ) / (c (i)  +  expval)); 
end; 

fa  =  fpd  +  (mu  -  ta) /s**2  ;  /*note  in  first  call,  theta=ta*/ 

/*first  part  worked...  and  matches  Dan's  cycle  0  value  for  fa=4. 30928*/ 
/*now  get  initial  estimate  of  fp  on  the  second  pass  through  the  data*/ 
/*second  part  worked...  and  matches  Dan's  cycle  1  value  for  fp=  -0.27936*/ 
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/*now  wrap  it  in  the  15  cycle  iteration  for  final  theta*/ 
do  krep  =  1  to  15; 
tp= (ta+tb) /2 ; 
fpd=0  ; 

do  i  =  1  to  nitems; 

expval=exp (a (i) *1 . 7* (tp-b (i) ) ) ;  /*note  in  second  call,  theta=tp*/ 
fpd  =  fpd  +  a(i)*1.7*(-l.  +  l./(l.  +  expval)  +  u(i)  - 
(c (i) *u (i) ) / (c (i)  +  expval)); 
end; 

fp  =  fpd  +  (mu  -  tp) /s**2  ;  /*note  in  second  call,  theta=tp*/ 
if  fp  =  0  then  stop  ; 
else  if  fa*fp  >  0  then 
do; 

ta=tp;  fa=fp; 
end; 

else  tb=tp; 
end; 

EI_theta= (ta+tb) /2 ; 
keep  ssn  El  theta  ; 
run; 

data  ScoreAS; 

set  WithParm  (keep=SSN  ASal-ASa25  ASbl-ASb25  AScl-ASc25  CAS01-CAS25 
ASitems) ; 
nitems=25; 

/*initialize  constants*/ 
ta=-4;  tb=4;  fpd=0;  mu=0;  s=l; 

/*set  arrays*/ 
array  a (25)  ASal-ASa25; 
array  b(25)  ASbl-ASb25; 
array  c(25)  AScl-ASc25; 
array  u(25)  CAS01-CAS25; 

/*now  get  initial  estimate  of  fa  on  the  first  pass  through  the  data*/ 
do  i  =  1  to  nitems; 

expval=exp (a (i) *1 . 7* (ta-b (i) ) ) ;  /*note  in  first  call,  theta=ta*/ 
fpd  =  fpd  +  a(i)*1.7*(-l.  +  l./(l.  +  expval)  +  u(i)  - 
(c  (i) *u (i) ) / (c (i)  +  expval)); 
end; 

fa  =  fpd  +  (mu  -  ta) /s**2  ;  /*note  in  first  call,  theta=ta*/ 

/*first  part  worked...  and  matches  Dan's  cycle  0  value  for  fa=4. 30928*/ 
/*now  get  initial  estimate  of  fp  on  the  second  pass  through  the  data*/ 
/*second  part  worked...  and  matches  Dan's  cycle  1  value  for  fp=  -0.27936*/ 
/*now  wrap  it  in  the  15  cycle  iteration  for  final  theta*/ 
do  krep  =  1  to  15; 
tp= (ta+tb) /2 ; 
fpd=0  ; 

do  i  =  1  to  nitems; 

expval=exp (a (i) *1 . 7* (tp-b (i) ) ) ;  /*note  in  second  call,  theta=tp*/ 
fpd  =  fpd  +  a(i)*1.7*(-l.  +  l./(l.  +  expval)  +  u(i)  - 
(c (i) *u (i) ) / (c (i)  +  expval)); 
end; 

fp  =  fpd  +  (mu  -  tp) /s**2  ;  /*note  in  second  call,  theta=tp*/ 
if  fp  =  0  then  stop  ; 
else  if  fa*fp  >  0  then 
do; 

ta=tp;  fa=fp; 
end; 
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else  tb=tp; 
end; 

AS_theta= (ta+tb) /2 ; 
keep  ssn  AS  theta  ; 
run; 

data  ScoreMC; 

set  WithParm  (keep=SSN  MCal-MCa25  MCbl-MCb25  MCcl-MCc25  CMC01-CMC25 
MCitems) ; 
nitems=25; 

/*initialize  constants*/ 
ta=-4;  tb=4;  fpd=0;  mu=0;  s=l; 

/*set  arrays*/ 
array  a (25)  MCal-MCa25; 
array  b(25)  MCbl-MCb25; 
array  c(25)  MCcl-MCc25; 
array  u(25)  CMC01-CMC25; 

/*now  get  initial  estimate  of  fa  on  the  first  pass  through  the  data*/ 
do  i  =  1  to  nitems; 

expval=exp (a (i) *1 . 7* (ta-b (i) ) ) ;  /*note  in  first  call,  theta=ta*/ 
fpd  =  fpd  +  a(i)*1.7*(-l.  +  l./(l.  +  expval)  +  u(i)  - 
(c  (i) *u (i) ) / (c (i)  +  expval)); 
end; 

fa  =  fpd  +  (mu  -  ta) /s**2  ;  /*note  in  first  call,  theta=ta*/ 

/*first  part  worked...  and  matches  Dan's  cycle  0  value  for  fa=4. 30928*/ 
/*now  get  initial  estimate  of  fp  on  the  second  pass  through  the  data*/ 
/*second  part  worked...  and  matches  Dan's  cycle  1  value  for  fp=  -0.27936*/ 
/*now  wrap  it  in  the  15  cycle  iteration  for  final  theta*/ 
do  krep  =  1  to  15; 
tp= (ta+tb) /2 ; 
fpd=0  ; 

do  i  =  1  to  nitems; 

expval=exp (a (i) *1 . 7* (tp-b (i) ) ) ;  /*note  in  second  call,  theta=tp*/ 
fpd  =  fpd  +  a(i)*1.7*(-l.  +  l./(l.  +  expval)  +  u(i)  - 
(c (i) *u (i) ) / (c (i)  +  expval)); 
end; 

fp  =  fpd  +  (mu  -  tp) /s**2  ;  /*note  in  second  call,  theta=tp*/ 
if  fp  =  0  then  stop  ; 
else  if  fa*fp  >  0  then 
do; 

ta=tp;  fa=fp; 
end; 

else  tb=tp; 
end; 

MC_theta= (ta+tb) /2 ; 
keep  ssn  MC  theta  ; 
run; 

data  ScoreAO; 

set  WithParm  (keep=SSN  AOal-AOa25  AObl-AOb25  AOcl-AOc25  CAO01-CAO25 
AOitems) ; 
nitems=25; 

/*initialize  constants*/ 
ta=-4;  tb=4;  fpd=0;  mu=0;  s=l; 

/*set  arrays*/ 

array  a (25)  AOal-AOa25; 

array  b(25)  AObl-AOb25; 
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array  c(25)  AOcl-AOc25; 
array  u(25)  CAO01-CAO25; 

/*now  get  initial  estimate  of  fa  on  the  first  pass  through  the  data*/ 
do  i  =  1  to  nitems; 

expval=exp (a (i) *1 . 7* (ta-b (i) ) ) ;  /*note  in  first  call,  theta=ta*/ 
fpd  =  fpd  +  a(i)*1.7*(-l.  +  l./(l.  +  expval)  +  u(i)  - 
(c (i) *u (i) ) / (c (i)  +  expval)); 
end; 

fa  =  fpd  +  (mu  -  ta) /s**2  ;  /*note  in  first  call,  theta=ta*/ 

/*first  part  worked...  and  matches  Dan's  cycle  0  value  for  fa=4. 30928*/ 
/*now  get  initial  estimate  of  fp  on  the  second  pass  through  the  data*/ 
/*second  part  worked...  and  matches  Dan's  cycle  1  value  for  fp=  -0.27936*/ 
/*now  wrap  it  in  the  15  cycle  iteration  for  final  theta*/ 
do  krep  =  1  to  15; 
tp= (ta+tb) /2 ; 
fpd=0  ; 

do  i  =  1  to  nitems; 

expval=exp (a (i) *1 . 7* (tp-b (i) ) ) ;  /*note  in  second  call,  theta=tp*/ 
fpd  =  fpd  +  a(i)*1.7*(-l.  +  l./(l.  +  expval)  +  u(i)  - 
(c (i) *u (i) ) / (c (i)  +  expval)); 
end; 

fp  =  fpd  +  (mu  -  tp) /s**2  ;  /*note  in  second  call,  theta=tp*/ 
if  fp  =  0  then  stop  ; 
else  if  fa*fp  >  0  then 
do; 

ta=tp;  fa=fp; 
end; 

else  tb=tp; 
end; 

AO_theta= (ta+tb) /2 ; 
keep  ssn  AO  theta  ; 

run;/*Merge  the  9  subtest  theta  estimates  by  ID  into  a  single  file*/ 
data  afct. theta; 

merge  ScoreGS  ScoreAR  ScoreWK  ScorePC  ScoreMK  ScoreEI  ScoreAS  ScoreMC 
ScoreAO; 
by  SSN; 

/*this  creates  the  standand  scores  from  EITHER  version  and  is  the  */ 
/*method  from  Dan's  PDF  doc,  chapter  4  plus  data  in  tables  2.1,  2.2*/ 
/*this  was  tested  against  Dan's  two  subject  example.  */ 

GSs  =  round (47 . 151224  +  10.527430  *  GS  theta,  1); 

ARs  =  round (47 . 667251  +  9.433283  *  AR  theta,  1); 

WKs  =  round ( 4 6 . 644 903  +  10.243609  *  WK  theta,  1); 

PCs  =  round (49 . 317406  +  10.097481  *  PC_theta,  1); 

MKs  =  round (47 . 936105  +  9.314591  *  MK  theta,  1); 

Els  =  round (47 . 865784  +  11.838647  *  El  theta,  1); 

ASs  =  round (47 . 914860  +  12.381293  *  AS  theta,  1); 

MCs  =  round (48 . 943491  +  10.677225  *  MC  theta,  1); 

AOs  =  round (50 . 649606  +  10.393213  *  AO_theta,  1); 

VEs  =  round (47 . 481779  +  6.861654*WK  theta  +  4.091172*PC  theta,  1); 
run; 

proc  print  data=afct . theta;  run; 

*/ 

/*this  does  a  table  lookup  for  the  AFQT,  based  on  Dan's  1997  norm 
publication* / 

/*it  works  for  any  test  version  where  the  standard  scores  are  on  the  */ 
/*1997  score  scale.  Uses  temp  arrays  with  the  look  up  values.  */ 


D-13 


*/ 

data  afct . theta_afqt; 
set  afct. theta  ; 


array  sums (161) 

temporary 

(109,  110, 

111, 

112,  113,  114, 

115, 

116, 

117 

,  118, 

119, 

120, 

121,  122, 

123 

124,  125, 
126,  127, 
141,  142, 

128, 

129,  130,  131, 

132, 

133, 

134, 

135, 

136, 

137, 

138,  139, 

140, 

143,  144, 
158,  159, 

145, 

146,  147,  148, 

149, 

150, 

151, 

152, 

153, 

154, 

155,  156, 

157, 

160,  161, 
175,  176, 

162, 

163,  164,  165, 

166, 

167, 

168, 

169, 

170, 

171, 

172,  173, 

174, 

177,  178, 
192,  193, 

179, 

180,  181,  182, 

183, 

184, 

185, 

186, 

187, 

188, 

189,  190, 

191, 

194,  195, 

196, 

197,  198,  199, 

200, 

201, 

202, 

203, 

204, 

205, 

206,  207, 

208, 

209,  210, 
211,  212, 
226,  227, 

213, 

214,  215,  216, 

217, 

218, 

219, 

220, 

221, 

222, 

223,  224, 

225, 

228,  229, 
243,  244, 

230, 

231,  232,  233, 

234, 

235, 

236, 

231 , 

238, 

239, 

240,  241, 

242, 

245,  246, 
260,  261, 

247, 

248,  249,  250, 

251, 

252, 

253, 

254, 

255, 

256, 

257,  258, 

259, 

262,  263, 

264, 

265,  266,  267, 

268, 

269) 

r 

array  afqts(161)  temporary 

(1,  2,  2, 

2,  2, 

2,  2,  2,  2,  2, 

3,  3 

,  3, 

3,  3 

,  3,  4 

,  4, 

4,  4, 

4,  4,  4, 

4,  4 

5,  5,  5,  5, 

6,  6 

1,  6,  6,  7 ,  7 , 

7,  7,  8,  8 

,  9, 

9,  9,  9,  10,  10 

,  11, 

11, 

11, 

12,  13 

,  13, 

14, 

15,  15,  16 

,  16 

17,  17,  18, 

19, 

19,  20,  21,  22, 

22,  23,  24 

,  25, 

25,  26,  27,  28 

,  29, 

30, 

31, 

32,  33 

,  34, 

35, 

35,  36,  38 

,  39 

40,  41,  42, 

43, 

44,  45, 

46,  47,  48 

,  49, 

50,  51,  52,  53 

,  54, 

55, 

56, 

57,  59 

,  60, 

61, 

62,  63,  64 

,  66 

67,  68,  69, 

70, 

70,  71, 

72,  73,  74 

,  75, 

76,  77,  78,  79 

,  80, 

81, 

82, 

82,  83 

,  84, 

85, 

85,  86,  87 

,  88 

88,  89,  90, 

90, 

91,  92, 

92,  93,  93 

,  93, 

94,  94,  95,  95 

,  95, 

96, 

96, 

96,  97 

,  97, 

97, 

97,  98,  98 

,  98 

98,  98,  99)  ; 
length  sum  afqt  3.0; 

sum=ARs+MKs+2 *VEs ;  /*sum  the  standard  scores...*/ 
af qt=0 ; 

if  sum<=109  then  afqt=l; 
if  sum>=269  then  afqt=99; 
do  i=l  to  161  until  (afqt>0) ; 
if  sum=sums(i)  then  af qt=af qts ( i )  ; 
end; 
run; 

proc  print  data=af ct . theta_af qt  ;  run; 
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Appendix  E: 

Web  AFCT  Equating  Study 
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Problem  -  Old  Technology 


A  computer-based  version  of  the  Armed  Services 
Vocational  Aptitude  Battery  (ASVAB)  is  administered  to 
Sailors  when  they  join  the  Navy 

To  improve  ASVAB  scores  while  in-service  (possibly  to 
qualify  for  a  different  rating),  they  receive  a  paper- 
based  version  of  the  Armed  Forces  Classification  Test 
(AFCT;  in-service  version  of  the  ASVAB) 

As  Navy  moves  more  training  and  testing  to  the 
Internet,  they  would  like  to  move  AFCT  administration 
to  the  Internet  as  well 

-  Less  time-consuming  for  examinees  as  well  as 
administrators 

-  Increased  security  of  the  test  items 
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Problem  -  Inaccurate  Scoring  and 
Recording 


AFCT  currently  does  not  use  item  response 
theory  so  cannot  be  scored  using  1997  scale 
scores 

Scores  are  hand-entered  into  personnel  files, 
increasing  the  potential  for  errors 


Does  creating  a  web-based  AFCT  impact 
results,  even  though  more  secure  and 
accurate? 
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Objective 


Create  a  web-based  version  of  AFCT  Form  19G 
Conduct  platform-effects  study 

-  Test  one  group  of  Sailors  with  the  paper- 
based  Form  19G  while  testing  a  matched 
group  using  the  web-based  version 

»  Match  on  gender  and  majority/minority 
status 

Determine  if  there  are  any  differences  in  scores 
overall  as  well  as  by  gender  and 
majority/minority  subgroups 
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AFCr  Description 


•  AFCT  is  the  version  of  the  ASVAB  administered  while  in 
the  service 

•  Test  consists  of  9  sections: 


Subject 

Number  of 
Questions 

Time 

Limit 

1 

General  Science  (GS) 

25 

11 

2 

Arithmetic  Reasoning  (AR) 

30 

36 

3 

Word  Knowledge  (WK) 

35 

11 

4 

Paragraph  Comprehension  (PC) 

15 

13 

5 

Mathematics  Knowledge  (MK) 

25 

24 

6 

Electronics  Information  (El) 

20 

9 

7 

Automotive  and  Shop  (AS) 

25 

11 

8 

Mechanical  Comprehension  (MC) 

25 

19 

9 

Assembling  Objects  (AO) 

25 

15 

Testing  conducted  31  July  -  13  September  2006 

Personnel  assigned  to  NTC  Great  Lakes  during 
testing  may  have  participated 

-  Majority  were  students  in  "A"  School  (tested  by 
duty  section)  or  in  INDOC  (after  last 
indoctrination  class  ended) 

-  Small  portion  were  Not  Under  Instruction 
(NUI)  while  two  were  staff  members  (E-6) 

Demographic  information  (gender  and 
majority/minority  status)  collected  through  sign- 
in  sheets 
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Conditions 


•  Majority  of  subjects  were  assigned  to  the  two 
conditions  based  upon  the  last  digit  of  SSN 

-  Those  with  even  last  digits  received  paper 
version  and  those  with  odd  received  web 
version 

•  Most  subjects  in  INDOC  (afternoon  sessions) 
were  assigned  to  the  web  condition  until  all 
computers  were  filled;  remaining  subjects  were 
assigned  to  the  paper  condition 

General  Form  19G  instructions  were  adapted  for 
experiment 

-  Web-based  instructions  were  provided  as  on¬ 
screen  text  that  the  subjects  read 


Paper-Based  Condition 


Testing  took  place  in  either  a  classroom  setting 
(individual  table/chairs  spaced  about  2  feet 
apart)  or  in  a  hangar  bay  (large  table  with  2 
chairs,  examinees  seated  in  every  other  chair) 

Standard  AFCT  paper  administration  procedures 
were  followed  throughout 

Two  proctors  were  available  to  read 
instructions,  pass  out  materials,  and  monitor 
the  testing  session 

Instructions  were  read  to  examinees  about  the 
voluntary  nature  of  the  testing  as  part  of  the 
standard  AFCT  instructions 
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Paper-Based  Condition  (continued) 


Each  participant  received  a  modified  AFCT 
Privacy  Act  Statement  indicating  that  scores 
would  only  count  if  they  were  better  than 
previous  ASVAB  scores 

Examinees  received  2  pencils,  2  pieces  of 
scratch  paper,  an  answer  sheet,  and  a  test  book 
while  the  instructions  were  read 

Test  administrators  read  instructions  at  the 
beginning  of  sections,  including  sample 
problem(s),  number  of  questions,  and  time  limit 
for  the  section 


Paper-Based  Condition  (continued) 


Examinees  read  questions  in  the  test  book,  with 
4  to  10  questions  per  pages  (depending  on 
section) 

Responses  were  marked  on  a  standard  "bubble" 
Scantron  answer  sheet 

Examinees  were  timed  by  test  administrators 

All  testing  materials  were  collected  before 
dismissal 


Web-Based  Condition  -  Similarities 
to  Paper 


Two  proctors  were  available  to  read 
instructions,  pass  out  materials,  and  monitor 
testing  throughout  session 

Examinees  were  read  instructions  about  the 
voluntary  nature  of  the  testing 

Each  received  a  modified  Privacy  Act  Statement 
indicating  that  scores  would  only  count  if  they 
were  better  than  previous  ASVAB  scores 

Each  received  2  pencils  and  2  pieces  of  scratch 
paper 
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Web-Based  Condition  -  Similarities 
to  Paper  (continued) 


Section  instructions  prefaced  each  section; 
instructions  were  not  timed  but  timing  started 
once  the  first  question  appeared 

Within  each  section,  the  examinee  could  go 
backwards  or  forwards,  and  skip  questions; 
subjects  could  not  return  to  previously 
completed  sections 

All  testing  materials  were  collected  as  each 
respondent  finished 
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Web-Based  Condition  -  Room 
Layout  and  Computers 


Testing  took  place  in  a  classroom  setting  (individual 
table/chair) 

-  Pairs  of  examinees  faced  each  other  with  laptop 
screens  between  them 

-  Each  pair  was  about  2  feet  distant  from  another  pair 
Compaq  Evo  N1020v  laptops  with  Windows  XP  and  IE 
6.x  were  used  as  testing  laptops 

Up  to  three  servers  (either  Windows  NT  or  2003)  were 
used  at  a  time,  with,  at  most,  23  testing  laptops 
connected  to  each 

-  Win  2003  server  took  longer  to  serve  up  pages,  but 
time  did  not  count  against  section  testing  time 


Web-Based  Condition  -  Room 
Layout 


I 


Web-Based  Condition  - 
Differences  from  Paper-Based 


•  While  waiting  to  begin  testing,  examinees  were 
allowed  to  play  games  on  the  computer  to  get 
used  to  the  koala-pad  mouse;  a  colored  dot  was 
included  on  the  left  mouse  button  for  ease  in 
identification 

•  Examinees  were  directed  to  follow  the 
instructions  on  the  computer  screen 

•  Sample  questions  were  provided  as  text  to  read 

•  Questions  and  answers  were  presented  in 
Internet  Explorer  (IE),  one  question  at  a  time 


I 
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Web-Based  AFCT 


Each  AFCT  section  was  programmed  in 
WebQuiz  XP 

-  Test  questions  and  data  stored  in  .mdb  files 

-  Question  display  handled  by  .asp  files 
Each  section  timed  through  WebQuiz 

Respondent  entered  SSN  at  beginning  of  first 
section  and  system  automatically  propagated 
SSN  to  succeeding  sections 

-  Respondent  could  not  go  back  to  previous 
section  without  restarting  entire  AFCT 


H 

tn 


Web-Based  AFCT  (continued) 


•  For  PC  section,  the  paragraph  was  presented 
along  with  the  first  question  about  that 
paragraph;  succeeding  questions  were  on 
separate  pages  and  examinee  could  go  back  to 
the  paragraph  if  needed 

•  Time  remaining  was  shown  at  the  bottom  left  of 
the  IE  window,  and  respondents  could  continue 
to  the  next  AFCT  section  on  their  own  (self- 
paced) 
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Web-Based  AFCT  -  Screen  Layout 
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Observed  Advantages  of  Web 
AFCT  Administration 


The  web  AFCT  is  self-paced,  so  Sailors  generally 
finished  sooner  than  on  paper 

-  Paper  testing  requires  a  minimum  of  2.5  hours  plus 
time  to  reacfinstructions 

-  At  least  half  of  the  web  examinees  were  done  within 
2  hours,  with  a  few  completing  in  1  hour 

Less  interaction  with  test  administrator  is  required  (in 
the  Fleet,  this  is  usually  another  Sailor  such  as  the 
Command  Career  Counselor) 

No  need  for  scanning  answer  forms  or  for  hand  entry  of 
data  -  reduces  error  due  to  incorrect  entry  or  scanning 

Increased  test  question  security 

Examinees  definitely  cannot  return  to  previous  sections 

No  chance  of  marking  two  answers  to  one  question 
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Observed  Disadvantages  of  Web 
AFCT  Administration 


All  Windows-based  computers  have  a  calculator 
installed,  but  calculators  are  not  allowed  on 
AFCT 

-  Calculators  need  to  be  removed  from 
Windows  program  menu  if  not  from  the 
computer  itself 

For  Paragraph  Comprehension  section,  one 
examinee  used  the  IE  Find  function  to 
electronically  find  key  words  in  the  paragraph 

-  Impact  of  this  would  depend  on  actual 
questions 
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Data  Inclusion  Rules 


Only  those  who  completed  (including  SSN)  and  signed  a 

Privacy  Act  Statement  are  included 

Staff  members  are  not  included 

-  Two  staff  members  took  the  web  version  of  the  test 

Only  the  first  set  of  answers  is  used  in  analyses 

-  Some  took  the  test  multiple  times  in  the  same  or 
different  format 

-  Two  people  took  the  test  in  the  same  week 

»  One  completed  one  session  completely  and  only 
the  first  section  of  the  second  session,  so  the 
complete  set  of  answers  is  used  in  analysis 

»  One  completed  both  versions  of  the  test  but  was 
not  included  because  it  was  unclear  which  version 
was  completed  first 


Data  Inclusion  Rules  (continued) 


Respondents  with  patterns  to  their  answers  (all  A,  ABCD 

repeated,  AABBCCDD  repeated,  etc.)  are  included 

-  Unclear  what  is  a  "pattern"  and  what  is  a  legitimate 
answer 

-  Likely  both  paper  and  web  respondents  displayed  this 
behavior 

Respondent  must  have  completed  at  least  half  of  a 

section  to  be  included  in  analyses  for  that  section 

-  Some  had  to  leave  before  completing  their  tests, 
while  others  chose  not  to  answer  any  questions 

-  List-wise  deletion  not  recommended  since  some 
legitimately  skipped  questions  or  ran  out  of  time  on 
the  section 
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Data  Used  in  Analyses 


•  Overall  data: 


Overall 

Male 

Female 

Majority 

Minority 

Paper (p) 

689 

547 

91 

412 

177 

Web  (w) 

762 

572 

111 

361 

229 

•  Ns  for  each  section  are  included  on  each  slide 


CO 

cc 
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Data  Scoring 


Test  scored  using  item  response  theory^ 

-  The  three  parameter  logistic  model  was  used 

-  Respondents  received  a  1  (if  answered 
correctly)  or  0  (if  answered  incorrectly  or 
blank) 

-  Theta  score  computed  from  parameters  using 
bisection  algorithm 

Theta  scores  translated  to  standard  scores 

(mean  50,  standard  deviation  10,  range  20-80) 

on  the  1997  ASVAB  normative  scale 

'  Segall,  D.  0.  (November  2005).  Item  Response  Theory  Scoring  Algorithms  for  AFCT  Forms  18F  and  19G:  1997  Score  Scale. 


Data  Scoring  (continued) 


Standard  score  ranges  vary  by  test 


Subject 

All  wrong 
answers 

All  correct 
answers 

1 

General  Science  (GS) 

19 

72 

2 

Arithmetic  Reasoning  (AR) 

24 

68 

3 

Word  Knowledge  (WK) 

15 

66 

4 

Paragraph  Comprehension  (PC) 

25 

62 

5 

Mathematics  Knowledge  (MK) 

27 

67 

6 

Electronics  Information  (El) 

23 

80 

7 

Automotive  and  Shop  (AS) 

23 

76 

8 

Mechanical  Comprehension  (MC) 

28 

75 

9 

Assembling  Objects  (AO) 

27 

66 

Verbal  (VE)  (Composite  of  PC  and  WK) 

16 

66 
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Linear  Equating 


Sponsor  requested  an  equating  study 

-  Linear  equating  chosen  because  of  the 
relatively  small  number  of  cases 

Based  upon  means  (p)  and  standard  deviation 
(o)  of  paper  and  web  tests,  lines  were 
computed  to  convert  between  the  two  versions: 

P={Opia„)*m  +(|Jp-((ap/aJ*iJ„)) 
w=(Ow/c'p)*P  +(Pw-((aw/ap)*Pp)) 
where  p  =  paper  and  w  =  web 
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Linear  Equating  Graphs 


Using  the  formulae,  equivalent  scores  were 
calculated  for  all  possible  scores  between  20 
and  80 

Scores  were  graphed  as  two  lines 

-  Red  line  shows  the  paper  score  for  a  given 
web  score 

-  Blue  line  shows  the  web  score  for  a  given 
paper  score 

Lines  that  are  closer  together,  especially  at  the 
scale  mean  of  50,  indicate  that  scores  from  the 
paper  and  web  tests  are  similar 


Arithmetic  Reasoning  (AR) 


80 

75 

70 

65 

g  60 

o 

55 

in 

+•> 

J  50 

I  45 

O’ 

lU 

40 

35 

30 

25 


-p=1.01w-.73 

w=.99p+.72 


20  25  30  35  40  45  50  55  60  65  70  75  80 

Actual  Score 


N 

Mean 

sd 

Paper (p) 

660 

51.77 

7.741 

Web  (w) 

739 

51.81 

7.640 

E-i6 


Paragraph  Comprehension  (PC) 


45  50  55 

Actual  Score 

*  Significant  difference  between  paper  and  web  (i^5.884,  />=.00). 


PC  scores  tend  to 
differ  at  the  low  end 
of  the  scale,  with 
paper  scores  higher 
than  web  scores. 


-p=.91w+6.98 

w=1.10g-7.68i 


N 

Mean 

sd 

Paper (p) 

647 

53.13 

7.004 

Web  (w)  * 

740 

50.79 

7.709 

■•4 


Verbal  (VE) 


VE  scores  are  based 
on  WK  and  PC 
scores. 


-p=1.04w-.72 

w=.96p+0.69 


20  25  30  35  40  45  50  55  60  65  70  75  80 

Actual  Score 

*  Significant  difference  between  paper  and  web  (f^3.149,  /?=.00). 


N 

Mean 

sd 

Paper (p) 

646 

53.82 

7.270 

Web  (w)  * 

739 

52.61 

7.013 

E-i8 


Mechanical  Comprehension  (MC) 


Consistent  difference 
between  paper  and 
web  for  MC,  with 
paper  being  higher. 


-p=1.01w+1.34| 

w=.99p-1.32 


N 

Mean 

sd 

Paper (p) 

646 

57.34 

8.932 

Web  (w)  * 

726 

55.32 

8.824 

*  Significant  difference  between  paper  and  web  (/^4.208,  /?=.00). 


Assembling  Objects  (AO) 


45  50  55 

Actual  Score 

*  Significant  difference  between  paper  and  web  (i^6.786,  />=.00). 


Large  difference 
between  AO  scores 
at  the  low  end  of  the 
scale. 


-p=.91w+8.43 

w=l.lQp-9.3Q 


80 


N 

Mean 

sd  1 

Paper (p) 

644 

55.10 

9.320 

Web  (w)  * 

721 

51.48 

10.281 

Armed  Forces  Qualification  Test 
(AFQT) 


41  51  61 

Actual  Score 

*  Significant  difference  between  paper  and  web  (f^2.695,  /?=.01). 


Armed  Forces 
Qualification  Test 
(AFQT)  is 

AR+MK+2VE;  scores 
range  from  1  to  99. 


-p=1.02w-2.30 

w=.99p-2.26 


N 

Mean 

sd 

Paper (p) 

646 

61.72 

22.019 

Web  (w)  * 

734 

58.54 

21.693 

E-20 


General  Science  (GS)  by  Gender 


Male 


Actual  Score 


-  p=1.10w-4.80 
w=.91p+4.38 


N 

Mean 

sd 

Paper (p) 

525 

54.62 

8.010 

Web  (w) 

552 

54.22 

7.309 

Female 


[  Large  difference 
I  for  females  on 
GS. 


-  p=.94w+6.35 
w=l.Q7p-6.79 


*  Significant  difference  between  paper  and  web  for  females  (/^2.908,  ^.00). 


N 

Mean 

sd 

Paper (p) 

90 

53.03 

7.257 

Web  (w)  * 

111 

49.92 

7.760 

Arithmetic  Reasoning  (AR)  by 
Gender 


Male 


Female 


E-21 


Word  Knowledge  (WK)  by  Gender 


Male 


Actual  Score 


-p=1.08w-4.15 

w=.93p+3.84 


N 

Mean 

sd 

Paper (p) 

515 

54.08 

7.784 

Web  (w) 

551 

53.88 

7.202 

Female 


N 

Mean 

sd 

Paper (p) 

87 

53.94 

6.900 

Web  (w)  * 

111 

51.32 

6.762 

*  Significant  difference  between  paper  and  web  for  females  (/^2.682,  ^.01). 


Paragraph  Comprehension  (PC)  by 
Gender 


E-22 


Verbal  (VE)  by  Gender 


40  45  50  55  60 

65  70  75  80 

■  20  25  30  35  4 

0  45  SO  55  6C 

Actual  Score 

-p=1.05w-1.73 

w=.95p+1.64 

p=.92w+7.66 

w=1.09p-8.34 

N 

Mean 

sd 

Paper (p) 

511 

54.08 

7.399 

Web  (w)  * 

550 

53.06 

7.035 

N 

Mean 

sd 

Paper (p) 

87 

54.52 

6.344 

Web  (w)  * 

no 

50.97 

6.902 

*  Significant  difference  between  paper  and  web  for  males  (/^2.302,  p=.Ql)  and  females  (^^3.714,  /7=.00). 


Mathematics  Knowledge  (MK)  by 
Gender 


Male 


Female 


Electronics  Information  (El)  by 
Gender 


Male 


Actual  Score 

-  p=1.00w+.01 
w=.99p-.Ql 


N 

Mean 

sd 

Paper (p) 

511 

59.22 

11.709 

Web  (w) 

545 

58.83 

11.634 

Female 


N 

Mean 

sd 

Paper (p) 

88 

52.90 

9.047 

Web  (w)  * 

111 

47.77 

8.590 

*  Significant  difference  between  paper  and  web  for  females  (/^4.087,  ^.00). 


Automotive  and  Shop  (AS)  by 
Gender 


Male 


Female 


E-24 


Mechanical  Comprehension  (MC) 
by  Gender 


Male 


-  p=1.01w+1.00 
w=.99p-.99 


N 

Mean 

sd 

Paper (p) 

510 

58.28 

8.912 

Web  (w)  * 

539 

56.47 

8.786 

Female 


-p=1.09w-.57 

w=.92p-.52 


*  Significant  difference  between  paper  and  web  for  males  (/^3.312,  /j=.00)  and  females  (/^3.837,  p=.00). 


65  70  75 


N 

Mean 

sd 

Paper (p) 

88 

53.98 

7.680 

Web  (w)  * 

109 

49.95 

7.033 

Assembling  Objects  (AO)  by 
Gender 


Armed  Forces  Qualification  Test 
(AFQT)  by  Gender 


Male 


N 

Mean 

sd 

Paper (p) 

511 

63.06 

21.972 

Web  (w)  * 

545 

60.40 

21.343 

Female 


-  p=.90w+15.54 
w=1.12p-17.34 


N 

Mean 

sd 

Paper (p) 

87 

62.62 

19.524 

Web  (w)  * 

no 

52.54 

21.787 

*  Significant  difference  between  paper  and  web  for  males  (/^1.993,  /j=.05)  and  females  {t=237S,  p=.Q0). 


General  Science  (GS)  by  Ethnicity 

- ► 


H 

to 


Majority 


-p=1.07w-3.29 

w=.94p+3.09 


N 

Mean 

sd 

Paper (p) 

399 

55.66 

7.469 

Web  (w) 

353 

55.24 

6.999 

Minority 


20  25  30  35  40  45  50  55  60  65  70  75  80 

Actual  Score 


*p=l.llw-3.93 

w=.90p+3.55 


N 

Mean 

sd 

Paper (p) 

166 

52.06 

8.227 

Web  (w) 

219 

50.51 

7.421 

E-26 


Arithmetic  Reasoning  (AR)  by 
Ethnicity 


Majority 


N 

Mean 

sd 

Paper (p) 

398 

52.95 

7.420 

Web  (w) 

351 

52.61 

7.776 

Minority 


N 

Mean 

sd 

Paper (p) 

163 

50.18 

8.221 

Web  (w) 

218 

51.10 

7.461 

Word  Knowledge  (WK)  by 
Ethnicity 


Majority 


Actual  Score 

*p=1.15w-7.92 
■  w=.87p+6.92 


N 

Mean 

sd 

Paper (p) 

394 

55.15 

7.289 

Web  (w) 

352 

55.06 

6.362 

50  SS  60  65  70 


Minority 


,U 

bj 

-'.V.  ' 

S  50 

? 

■  ■ 

.  •  • 

4u 

35 

30 

^ ^ ^ ^ ^ ^ ^ ^ ^ 

20  25  30  35 


45  SO  55  60  65  70  75 

Actual  Score 


-p=1.03w-.03 

w=.97p+.03 


N 

Mean 

sd 

Paper (p) 

159 

52.01 

8.183 

Web  (w) 

219 

50.49 

7.940 

E-27 


Paragraph  Comprehension  (PC)  by 
Ethnicity 


N 

Mean 

sd 

Paper (p) 

391 

54.04 

6.817 

Web  (w)  * 

352 

51.74 

7.539 

Minority 

Large  difference  for 
both  majority  and 
minority  on  PC. 


40  45  SO  5S  60 

Actual  Score 

65  70  75  BO 

20  25  30  35 

40  45  SO  55  60 

Actual  Score 

65  70  75  80 

—  p=.90vy+7.26 
w=l.llp-8.03 

p=.90w+8.49 

w=l.llp-9.48 

N 

Mean 

sd 

Paper (p) 

158 

52.39 

7.127 

Web  (w)  * 

218 

49.04 

7.960 

*  Significant  difference  between  paper  and  web  for  majority  (^^4.367,  /j=.00)  and  minority  (^^4.207,  p=.00). 


Verbal  (VE)  by  Ethnicity 

Majority 


Minority 


E-28 


Mathematics  Knowledge  (MK)  by 
Ethnicity 


Majority 


Actual  Score 


-p=1.04w-1.62 
•  w=.96p+1.56 


N 

Mean 

sd 

Paper (p) 

392 

54.60 

7.003 

Web  (w) 

350 

53.83 

6.705 

Minority 


Actual  Score 


-p=1.12w-5.55 
•  w=.90p+4.97 


N 

Mean 

sd 

Paper (p) 

158 

52.50 

7.173 

Web  (w) 

219 

52.02 

6.427 

Electronics  Information  (El)  by 
Ethnicity 


Majority 


Minority 


Automotive  and  Shop  (AS)  by 
Ethnicity 


Majority 


Actual  Score 


-  p=.97w+1.42 
w=1.03p-1.47 


N 

Mean 

sd 

Paper (p) 

390 

53.95 

9.401 

Web  (w) 

346 

54.15 

9.692 

Minority 


N 

Mean 

sd 

Paper (p) 

158 

46.68 

9.216 

Web  (w) 

217 

46.16 

8.769 

Mechanical  Comprehension  (MC) 
by  Ethnicity 


Majority 


-p=1.03w-.77 

w=.96p+.75 


N 

Mean 

sd 

Paper (p) 

390 

58.92 

8.725 

Web  (w) 

345 

57.51 

8.405 

Minority 

Large  difference  for 
minority  on  MC. 


N 

Mean 

sd 

Paper (p) 

158 

54.66 

8.688 

Web  (w)  * 

215 

51.73 

8.569 

*  Significant  difference  between  paper  and  web  for  minority  (/^3.244,  ;>=.00). 


Assembling  Objects  (AO)  by 
Ethnicity 


Majority 


-p=.91w+8.58 

w=1.10p-9.47 


N 

Mean 

sd 

Paper (p) 

390 

55.87 

9.271 

Web  (w)  * 

343 

52.15 

10.225 

Minority 


-p=.94w+6.27 

w=1.06p-6.63 


N 

Mean 

sd 

Paper (p) 

158 

53.77 

9.860 

Web  (w)  * 

213 

50.28 

10.437 

*  Significant  difference  between  paper  and  web  for  majority  (f=5.165,  /j=.00)  and  minority  (f=3.260,  p=.00). 


Armed  Forces  Qualification  Test 
(AFQT)  by  Ethnicity 


Majority 


Minority 


Summary 


Web  AFCT  as  programmed  was  stable 

-  Only  issue  encountered  related  to  lost 
connectivity  of  laptops  to  the  server 

For  overall  scores,  paper  scores  tend  to  be 
slightly  higher,  with  noticeable  differences  for: 

-  PC  (5  points  at  lowest  end  of  scale) 

-  AO  (about  7  points  at  lowest  end  of  scale) 

Scores  for  males  are  similar  to  overall  scores, 
with  noticeable  differences  again  for  PC  and  AO 


H 

tn 


Summary  (continued) 


•  Scores  for  females  showed  larger  differences, 
perhaps  due  to  the  low  number  of  women 

-  Paper  scores  tended  to  be  higher,  with 
differences  for  PC  and  AO  being  10  points  or 
more 

•  Both  majority  and  minority  showed  differences, 
most  noticeable  again  on  PC  and  AO  (5  points 
or  more) 


Recommendations 


Record  scores  of  study  examinees,  if  better 
than  their  previous  scores 

Determine  a  concept  of  operations  for 
administering  the  Web  AFCT  to  the  Fleet 

-  Decide  if  hosting  through  NKO  is  an  option 

-  Determine  how  to  automatically  update 
personnel  files  and  incorporate  IRT  scoring 

-  Find  solution  to  administration  aboard 
ship/submarine 


65 


Recommendations  (continued) 


Brief  MAPWG  on  results 

-  Determine  if  any  point  difference  (e.q.,  2 

liqibre  i 


or  less)  could  be  considered  negligil 
scores 


pts 


in 


-  Determine  how  web  scores  should  be 
recorded  in  databases  (i.e.,  record  the  paper 
equivalent  score  or  the  actual  web  score) 

Repeat  study  to  test  possible  improvements 

-  Change  font  type  and  size  for  PC 

-  Use  grease  pencils  on  monitor  or  on 
transparency  overlay  for  MC/AO 

Repeat  or  expand  study  to  increase  dataset 


Project  Team 

Customer  Team 

NPRSTTeam 

Selection  and  Classification  (N132) 

Zannette  Uriell,  Principal  Investigator 

Dr.  Steve  Watson 

Evangeline  Clewis,  SOC  programming 

Dr.  Lisa  Mills 

David  L.  Alderton,  Ph.D.,  programming 
Paul  Rosenfeld,  Ph.D. 

NPRST  Proctors 

Zannette  Uriell 

Paul  Rosenfeld,  Ph.D. 

TSC  Great  Lakes  Team 

Rosemary  Schultz,  Ph.D. 

GSCM  Christopher  K.  Knerr 

Terri  Ferraro 

1 — 

O) 

FCCS  Douglas  S.  Berry 

Perry  Pena 

ETCS  Ryan  N.  Biggin 

Geoff  Patrissi 

K3 

FCC  Christopher  D.  Polk 

Carol  Newell 

■3 

GSMl  Jose  Barraza 

Gerry  Wilcove 

Bd 

MMl  Dae  S.  Kim 

Joyce  Alexander 

■c/  EMI  Lawrence  Palacio 

Olivia  Puentes 

H 

■p  FC2  William  T.  Griseto 

Rodney  Myers  e? 

Background 


NPRST  conducted  testing  of  web-based  AFCT  at 
NAVSTA  Great  Lakes  in  2006 

-  Project  sponsored  by  Dr.  Watson  (N132G) 

Personnel  generally  assigned  to  two  groups 
(paper-based  administration  or  web-based 
administration)  based  upon  last  digit  of  SSN 

-  Due  to  room  limitations,  respondents  in 
certain  sessions  were  more 
based  administration,  regardless  of  SSN 

At  MAPWG's  request,  need  to  determine  if  two 
administration  groups  were  different  prior  to 
testing,  based  upon  ASVAB  record  scores 


ikely  to  take  web- 


Data  and  Analyses 


ASVAB  record  data  extracted  for  personnel  with 
useable  scores 

A  few  had  record  data  scores  of  0;  those  people 
were  removed  from  both  record  and  testing 
data  for  these  analyses  so  that  their  0  score 
would  not  impact  computation  of  means 

f-test  of  independent  means  conducted, 
comparing  the  web-based  administration  group 
to  the  paper-based  administration  group,  both 
prior  to  testing  and  with  testing  results 


Record  Data  -  Overall 


1  Those  who  took  web- 
based  AFCT 

Those  who  took  paper- 
based  AFCT 

■nT 

Mean 

SD 

N 

Mean 

SD 

GS 

723 

54.09 

7.431 

657 

54.85 

7.446 

AR 

722 

54.76 

6.303 

654 

54.90 

6.485 

WK  * 

724 

52.39 

6.995 

645 

53.43 

6.864 

PC* 

722 

53.81 

6.079 

642 

54.71 

5.799 

MK 

722 

55.93 

5.396 

644 

56.26 

5.586 

El  * 

718 

54.69 

8.637 

643 

56.03 

8.386 

AS 

714 

51.88 

9.115 

642 

52.99 

8.511 

MC 

707 

55.67 

8.364 

642 

56.49 

8.078 

AO 

629 

57.17 

6.974 

584 

57.63 

7.318 

* 

LJJ 

> 

722 

53.09 

6.295 

641 

54.11 

6.103 

Iafqt 

716 

64.41 

18.065 

642 

66.83 

17.778 

*  Indicates  significant  difference  (p  <  .01)  between  the  two  groups. 


T 

P 

-1.896 

.058 

-.406 

.685 

-2.770 

.006 

-2.789 

.005 

-.922 

.357 

-2.897 

.004 

-2.310 

.021 

-1.828 

.068 

-1.121 

.263 

-3.029 

.003 

-2.483 

.013 

Testing  Data  -  Overall 


Those  who  took  web- 
based  AFCT 

Those  who  took 
paper-based  AFCT 

N 

Mean 

SD 

N 

Mean 

SD 

T 

P 

GS 

723 

53.34 

7.545 

657 

54.02 

8.015 

-1.623 

.105 

AR 

722 

51.88 

7.629 

654 

51.79 

7.739 

.217 

.828 

WK 

724 

53.42 

7.155 

645 

53.77 

7.627 

-.876 

.381 

PC  * 

722 

50.87 

7.679 

642 

53.13 

7.026 

-5.646 

.000 

MK 

722 

55.93 

5.396 

644 

56.26 

5.586 

-1.513 

.130 

El 

718 

56.74 

11.868 

643 

57.77 

11.644 

-1.613 

.107 

AS 

714 

50.81 

10.234 

642 

51.58 

9.861 

-1.407 

.160 

MC* 

707 

55.37 

8.854 

642 

57.34 

8.939 

-4.063 

.000 

> 

o 

* 

629 

51.86 

10.202 

584 

55.17 

9.233 

-5.909 

.000 

< 

m 

* 

722 

52.69 

7.008 

641 

53.82 

7.290 

-2.915 

.000 

Iafqt 

716 

58.85 

21.628 

642 

61.76 

22.011 

-2.454 

.014 

*  Indicates  significant  difference  (p  <  .01)  between  the  two  groups. 
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Record  Data  -  Males 


1  Those  who  took  web- 
based  AFCT 

Those  who  took  paper- 
based  AFCT 

Mean 

SD 

N 

Mean 

SD 

GS 

548 

55.12 

6.945 

521 

55.81 

7.034 

AR 

547 

55.48 

6.201 

517 

55.37 

6.354 

WK 

549 

52.93 

7.005 

511 

53.80 

6.703 

PC 

548 

54.07 

6.121 

508 

54.89 

5.872 

MK 

547 

56.00 

5.494 

509 

54.46 

5.735 

El 

543 

56.16 

8.334 

508 

57.28 

8.078 

AS 

541 

53.67 

8.693 

507 

54.35 

8.172 

MC 

535 

57.18 

7.936 

507 

57.64 

7.912 

AO 

473 

57.63 

6.853 

462 

57.92 

7.420 

VE 

548 

53.54 

6.292 

508 

54.44 

5.996 

AFQT 

542 

65.99 

18.013 

508 

68.02 

17.645 

T 

P 

-1.614 

.107 

.286 

.775 

-2.063 

.039 

-2.218 

.027 

-1.331 

.183 

-2.210 

.027 

-1.303 

.193 

-.937 

.349 

-.633 

.527 

-2.376 

.018 

-1.843 

.066 

Testing  Data  -  Males 


Those  who  took  web- 
based  AFCT 

Those  who  took 
paper-based  AFCT 

N 

Mean 

SD 

N 

Mean 

SD 

T 

P 

GS 

548 

54.26 

7.298 

521 

54.64 

8.015 

-.811 

.417 

AR 

547 

52.38 

7.660 

517 

52.20 

7.786 

.380 

.704 

WK 

549 

53.89 

7.213 

511 

54.06 

7.796 

-.369 

.712 

PC  * 

548 

54.10 

7.742 

508 

53.31 

7.042 

-4.840 

.000 

MK 

547 

53.51 

6.702 

509 

54.07 

7.086 

-1.320 

.187 

El 

543 

58.84 

11.653 

508 

59.19 

11.669 

-.486 

.627 

AS 

541 

52.70 

9.896 

507 

54.35 

8.172 

-.476 

.634 

MC* 

535 

56.51 

8.789 

507 

58.29 

8.912 

-3.245 

.001 

* 

o 

< 

743 

52.39 

10.246 

462 

55.18 

9.394 

-4.337 

.000 

VE 

548 

53.07 

7.045 

508 

54.08 

7.411 

-2.270 

.023 

AFQT 

542 

60.45 

21.384 

508 

63.09 

21.944 

-1.974 

.049 

*  Indicates  significant  difference  (p  <  .01)  between  the  two  groups. 
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Record  Data  -  Females 


1  Those  who  took  web- 
based  AFCT 

Those  who  took  paper- 
based  AFCT 

■nT 

Mean 

SD 

N 

Mean 

SD 

GS 

no 

49.99 

7.933 

90 

52.10 

7.801 

AR 

no 

52.27 

5.809 

90 

53.12 

7.166 

WK* 

no 

50.25 

6.664 

87 

53.15 

7.313 

PC 

109 

52.53 

5.690 

88 

54.61 

5.621 

MK 

no 

55.75 

5.087 

88 

56.20 

4.797 

El 

no 

48.81 

6.798 

88 

50.02 

7.171 

AS 

109 

43.80 

5.434 

88 

46.06 

6.970 

MC  * 

108 

48.83 

5.783 

88 

51.42 

6.612 

AO 

100 

55.56 

6.450 

79 

57.11 

6.528 

* 

LJJ 

> 

109 

51.04 

6.901 

87 

54.52 

6.344 

Iafqt 

109 

58.57 

16.727 

87 

64.95 

18.197 

*  Indicates  significant  difference  (p  <  .01)  between  the  two  groups. 


T 

P 

-1.885 

.061 

-.927 

.355 

-2.905 

.004 

-2.565 

.011 

-.634 

.527 

-1.215 

.226 

-2.557 

.011 

-2.924 

.004 

-1.588 

.114 

-2.971 

.003 

-2.551 

.012 

Testing  Data  -  Females 


Those  who  took  web- 
based  AFCT 

Those  who  took 
paper-based  AFCT 

N 

Mean 

SD 

N 

Mean 

SD 

T 

P 

GS  * 

no 

50.00 

7.748 

90 

53.03 

7.257 

-2.831 

.005 

AR 

no 

49.96 

7.212 

90 

51.67 

6.722 

-1.720 

.087 

WK 

no 

51.38 

6.757 

87 

53.94 

6.900 

-2.616 

.010 

PC  * 

109 

50.06 

7.366 

88 

54.53 

5.628 

-4.693 

.000 

MK 

no 

51.95 

5.927 

88 

53.61 

6.510 

-1.874 

.062 

El  * 

no 

47.84 

8.596 

88 

52.90 

9.047 

-4.021 

.000 

AS  * 

109 

42.22 

6.148 

88 

44.94 

6.464 

-3.017 

.003 

MC* 

108 

49.98 

7.060 

88 

53.98 

7.680 

-3.792 

.000 

> 

o 

* 

100 

49.99 

9.615 

79 

56.37 

8.426 

-4.652 

.000 

< 

m 

* 

109 

51.04 

6.901 

87 

54.52 

6.344 

-3.635 

.000 

1 AFQT  * 

109 

52.77 

21.748 

87 

62.62 

19.524 

-3.295 

.001 

*  Indicates  significant  difference  (p  <  .01)  between  the  two  groups. 
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Record  Data  -  Majority 


1  Those  who  took  web- 
based  AFCT 

Those  who  took  paper- 
based  AFCT 

■nT 

Mean 

SD 

N 

Mean 

SD 

GS 

352 

56.03 

6.361 

395 

56.65 

6.269 

AR 

350 

55.92 

6.019 

394 

55.90 

6.358 

WK 

351 

54.20 

6.358 

390 

54.58 

6.573 

PC 

351 

54.92 

5.829 

388 

55.34 

5.579 

MK 

349 

56.13 

5.733 

389 

56.56 

5.590 

El 

346 

56.64 

7.879 

388 

57.68 

7.605 

AS 

345 

55.23 

8.288 

387 

55.21 

7.806 

MC 

344 

58.16 

7.566 

387 

58.26 

7.713 

AO 

308 

57.92 

6.768 

356 

58.42 

7.182 

VE 

351 

54.71 

5.716 

388 

55.10 

5.836 

AFQT 

348 

68.94 

16.985 

388 

70.00 

16.544 

*  Indicates  significant  difference  (p  <  .01)  between  the  two  groups. 


T 

P 

-1.340 

.181 

.044 

.965 

-.798 

.425 

-1.000 

.317 

-1.031 

.303 

-1.818 

.069 

.034 

.973 

-.177 

.860 

-.919 

.359 

-.916 

.360 

-.857 

.392 

Testing  Data  -  Majority 


Those  who  took  web- 
based  AFCT 

Those  who  took 
paper-based  AFCT 

N 

Mean 

SD 

N 

Mean 

SD 

T 

P 

GS 

352 

55.23 

7.004 

395 

55.70 

7.465 

-.884 

.377 

AR 

350 

52.61 

7.788 

394 

52.97 

7.412 

-.646 

.519 

WK 

351 

55.07 

6.369 

390 

55.14 

7.304 

-.138 

.890 

PC  * 

351 

51.74 

7.549 

388 

54.05 

6.833 

-4.366 

.000 

MK 

349 

53.83 

6.714 

389 

54.63 

6.980 

-1.583 

.114 

El 

346 

59.53 

11.616 

388 

59.66 

11.256 

-.154 

.878 

AS 

345 

54.12 

9.691 

387 

53.94 

9.411 

.255 

.799 

MC 

344 

57.51 

8.418 

387 

58.93 

8.723 

-2.233 

.026 

> 

o 

* 

308 

52.63 

10.134 

356 

56.08 

9.085 

-4.625 

.000 

VE 

351 

54.13 

6.394 

388 

55.09 

7.052 

-1.931 

.054 

AFQT 

348 

62.88 

20.632 

388 

66.28 

20.346 

-2.248 

.025 

*  Indicates  significant  difference  (p  <  .01)  between  the  two  groups. 


Record  Data  -  Minority 


1  Those  who  took  web- 
based  AFCT 

Those  who  took  paper- 
based  AFCT 

■nT 

Mean 

SD 

N 

Mean 

SD 

GS 

216 

51.00 

7.623 

166 

52.57 

7.720 

AR 

217 

53.52 

6.317 

163 

53.34 

6.757 

WK* 

218 

49.61 

6.875 

159 

52.21 

6.842 

PC  * 

216 

52.06 

6.084 

158 

54.10 

6.011 

MK  * 

218 

55.82 

4.997 

158 

56.28 

5.746 

El 

217 

51.99 

8.319 

158 

53.27 

8.979 

AS 

216 

47.27 

7.944 

158 

48.74 

8.161 

MC 

212 

52.06 

7.930 

158 

53.50 

7.685 

AO 

188 

55.88 

6.986 

141 

56.66 

7.294 

* 

LJJ 

> 

217 

50.57 

6.231 

158 

53.12 

6.134 

1  AFQT  * 

215 

58.01 

17.647 

158 

63.32 

19.258 

*  Indicates  significant  difference  (p  <  .01)  between  the  two  groups. 


T 

P 

-1.984 

.048 

.267 

.790 

-3.962 

.000 

-3.219 

.001 

6.864 

.000 

-1.423 

.156 

-1.747 

.081 

-1.751 

.081 

-.983 

.326 

-3.939 

.000 

-2.762 

.006 

Testing  Data  -  Minority 


Those  who  took  web- 
based  AFCT 

Those  who  took 
paper-based  AFCT 

N 

Mean 

SD 

N 

Mean 

SD 

T 

P 

GS 

216 

50.61 

7.420 

166 

52.06 

8.227 

-1.806 

.072 

AR 

217 

51.14 

7.447 

163 

50.18 

8.221 

1.189 

.235 

WK 

218 

50.52 

7.946 

159 

52.01 

8.183 

-1.943 

.053 

PC  * 

216 

49.09 

7.976 

158 

52.39 

7.127 

-4.132 

.000 

MK 

218 

52.03 

6.441 

158 

52.50 

7.173 

-.623 

.534 

El 

217 

52.66 

11.507 

158 

55.33 

11.601 

-2.211 

.028 

AS 

216 

46.19 

8.776 

158 

46.68 

9.216 

-.522 

.602 

MC* 

212 

51.78 

8.580 

158 

54.66 

8.688 

-3.177 

.002 

> 

o 

* 

188 

50.15 

10.448 

141 

53.71 

9.854 

-3.133 

.002 

< 

m 

* 

217 

49.96 

7.689 

158 

52.39 

7.605 

-3.036 

.003 

lAFQT 

215 

51.86 

22.497 

158 

56.46 

23.062 

-1.931 

.054 

*  Indicates  significant  difference  (p  <  .01)  between  the  two  groups. 


RST 


Summary 


The  two  groups  were  not  equivalent  overall  when  they 
participated  in  the  web-based  AFCT  effects  study 

-  Male  and  majority  subgroups  were  equivalent  based 
on  record  data,  but  females  and  minority  were  not 

-  In  the  overall  group,  those  who  took  the  paper-based 
version  were  already  slightly  better  in  WK  and  PC 
(and,  thereby,  VE)  and  El 

Effects  study  results  for  overall  group  show  differences 
in  PC,  MC,  AO,  and  VE 

-  Original  differences  between  the  two  groups  for  WK 
disappear 

-  MC  and  AO  show  differences  that  are  not  directly 
attributable  to  any  pre-existing  differences  between 
the  two  groups 


Appendix  F: 

Detailed  Results  by  Test  Question 


F-o 


Table  F-1 

GS  for  all  respondents 
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Table  F-2 

GS  for  male  respondents 
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Table  F-3 

GS  for  female  respondents 
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Table  F-4 

GS  for  majority  respondents 


F-4 


Table  F-5 

GS  for  minority  respondents 
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Significant  difference  between  paper  and  web  (p  <  .01). 
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Table  F-9 

AR  for  majority  respondents 
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Table  F-12 

WK  for  male  respondents 
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Table  F-13 

WK  for  female  respondents 
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Table  F-14 

WK  for  majority  respondents 
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Table  F-21 
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Table  F-22 

MK  for  male  respondents 
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Table  F-23 

MK  for  female  respondents 
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Table  F-24 

MKfor  majority  respondents 
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Table  F-25 

MKfor  minority  respondents 
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MKfor  minority  respondents 
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Table  F-26 

El  for  all  respondents 
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Significant  difference  between  paper  and  web  (p  <  .01). 


Table  F-27 

El  for  male  respondents 


F-42 


Table  F-28 

El  for  female  respondents 


F-43 


Significant  difference  between  paper  and  web  (p  <  .01). 
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AS20  240  45  297  55  6  1  228  45  274  55  8  2  1.766  2  0.414 

AS21  219  41  318  59  6  1  207  41  296  59  7  1  1.775  2  0.412 
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Table  F-33 

AS  for  female  respondents 
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Table  F-33 

AS  for  female  respondents 
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Table  F-34 

AS  for  majority  respondents 
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AS  for  majority  respondents 
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Table  F-35 

AS  for  minority  respondents 
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Table  F-35 

AS  for  minority  respondents 
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Table  F-36 

MC  for  all  respondents 
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MC  for  all  respondents 
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Table  F-37 

MC  for  male  respondents 
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Table  F-37 

MC  for  male  respondents 
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Table  F-38 

MC  for  female  respondents 
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Table  F-38 

MC  for  female  respondents 
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Table  F-39 

MC  for  majority  respondents 
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Table  F-39 

MC  for  majority  respondents 
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Table  F-40 

MC  for  minority  respondents 


F-64 


Table  F-40 

MC  for  minority  respondents 
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Table  F-41 

AO  for  all  respondents 
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Table  F-41 

AO  for  all  respondents 
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Table  F-42 

AO  for  male  respondents 
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Table  F-42 

AO  for  male  respondents 
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Table  F-43 

AO  for  female  respondents 
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Table  F-43 

AO  for  female  respondents 
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Table  F-44 

AO  for  majority  respondents 
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Table  F-44 

AO  for  majority  respondents 
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Table  F-45 

AO  for  minority  respondents 
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Table  F-45 

AO  for  minority  respondents 
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